[Docs] Add RL example for RDT, vLLM, FSDP2, and GRPO. #58314
Conversation
Signed-off-by: Ricardo Decal <rdecal@anyscale.com>
|
|
||
|
|
||
| @ray.remote(name=REGISTRY_NAME) | ||
| class RayObjectRefRegistry: |
There was a problem hiding this comment.
This is my workaround for sending the RDT ObjectRef to the vLLM workers. collective_rpc() is unable to send the ObjectRef because of the way it serializes objects: https://gist.github.com/crypdick/8bd703085f5c8f8b2f4d2def58bac516
| self.optim.step() | ||
| else: | ||
| print( | ||
| "[WARNING] Skipping optimizer step due to zero gradients - all samples likely have same reward", |
There was a problem hiding this comment.
Without this, after the first training step the Generator state collapses and outputs nothing but end of sentence tokens: <s><s><s><s><s><s><s>. One simple way to fix this is to give smoother rewards, i.e. instead of giving all wrong answers 0 reward, giving a bit more reward if the answer is closer to the target.
|
This pull request has been automatically marked as stale because it has not had You can always ask for help on our discussion forum or Ray's public slack channel. If you'd like to keep this open, just leave any comment, and the stale label will be removed. |
github-actions
bot
added
the
stale
| for name, weight in state_dict.items(): | ||
| # FIXME: Qiaolin, remove deepcopy once the bug with sending the same weights multiple times is fixed. | ||
| name_weight = (name, weight) | ||
| name_weight = copy.deepcopy(name_weight) |
There was a problem hiding this comment.
Do we still need to use deepcopy here?
There was a problem hiding this comment.
@Qiaolin-Yu It was necessary at the time I opened the PR, I am not sure if it has been fixed since then
github-actions
bot
added
unstale
Description
Demonstrates RDT in using RL post-training of an LLM against the GSM8K dataset.
Related issues
Companion PR to #57961