🚀 Enhance GRPO VLLM server from sync to async and accelerate training

[binary-husky]

1. Change VLLM Server from Sync to Async
   • if is_async=True:
     client first call generate (non-blocking), then after a while call get_future (with identical arguments) to get result
   • if is_async=False:
     client automatically call get_future inside generate, blocking further execution before the generation is complete

2. Speed up grpo_trainer 1.5x faster by submitting N=gradient_accumulation_steps batches, so that training and vllm generation can run in parallel!
   However, I have to admit that this piece of code is not elegant enough, remove them if they disqualifies.

3. I leave some room by adding a RolloutEngine in trl.scripts.vllm_serve, for more sophisticated vllm inference functionality, trying to support lm_generate > MCP tool_call > lm_generate > another MCP tool_call > ..., but not complete yet.

4. add vllm_server_nccl_port in config (previously cannot change default)

[binary-husky]

oh, there is another detail worth mentioning:

I add a version param, self.version += 1 whenever update_model_params is called.

at server side, I add some lines to ensure there are no on-going generation with some async sleep logic

[fabianlim]

@binary-husky the speedups you posted look great, though I have a question on how you parallelize the computation. THe picture shows a data dependency between roll outs and model training (and vllm update).

• are you saying that within gradient accumulation steps the rollouts do not change?
• the completion_ids are futures, are you saying the will return enough rollouts for you to complete the grad accum step?

In other words, this achieve parallization within grad accum steps, and works only if the grad accum > 1?

[binary-husky]

@fabianlim Yes, works only if the grad acc step > 1.

[tcapelle]

We should make this happen! The new Mistral reasoning model uses a pipeline like this one.

[tcapelle]

Shouldn't populating _buffered_inputs asynchronously be enough?