Single-node 8xH100 examples for 32B models for long-context training (near 32k tokens regime)

[vadimkantorov]

Hi!

Are there any examples of realistic RL-tuning of 32B model with large model len (~32k with ignore_eos=True to ensure it can run near the maximum-length responses)?

I found

Trinity-RFT/examples/dapo_math/dapo.yaml

Line 5 in 919a5a2

model_path: ${oc.env:TRINITY_MODEL_PATH,Qwen/Qwen2.5-32B-Instruct}

but it's suspiciously uses tp=1 sp=1. Does 32B training fits a single node in this experiment?

Or what is the actual number of response tokens in this experiment?

What would be your advice on setting tp and sp for 32B models? (both for single-node and for multi-node) Should we try sp=2, tp=2, tp=2xsp=2 configs?

Or should we just try sp=8 first?

Thanks!

---

Sorry, got wrong the label. This should not be marked as a bug

[chenyushuo]

We tested the Qwen3-30B-A3B model on a system with 8 H20 GPUs—using 6 for training and 2 for exploration. Each GPU has around 97,871 MiB of memory. Even after reducing the maximum token length per GPU (setting ppo_max_token_len_per_gpu to 1600), which forces a very small batch size of just 1 sample per GPU (ppo_micro_batch_size_per_gpu=1), we still ran into out-of-memory (OOM) errors. We also tried using offloading techniques, but they didn't solve the issue.

Based on my testing in #231, the Qwen3-30B-A3B model simply can't run on a single machine, and this likely applies to the standard 32B model as well. My suggestion is to either scale out across multiple machines or switch to a smaller model that fits your hardware constraints.

[vadimkantorov]

Have you used any sp/tp? cpu offloading?

[vadimkantorov]

E.g. should work to set fsdp_size: 8 and ulysses_sequence_parallel_size: 8? To ensure sharding of MLP's, FF's and Ulysses attention?

Curious, is it possible to also TP-shard LayerNorm's as proposed in https://docs.pytorch.org/tutorials/intermediate/TP_tutorial.html#how-tensor-parallel-works for fsdp2? Need to provide custom tp config and invoke manually parallelize_module? I found that torchtune does exactly that, but not Verl. Does it mean that Verl does not support TP (besides crude FSDP sharding)?

How many 8-GPU nodes did you have to use to train a 32B model on long contexts?

32B model weights (params + optimizer states in bf16) should take 24Gb per GPU (out of 80Gb) when fully-sharded (and especially if param/states RAM-offloading is used). Seems should be sufficient space for activations (when using checkpointing) - if TP+SP (in PyTorch native style) + CP (Ulysses-style) is used - at least using RAM activation offloading?

---

Also, does Verl/Trinity maintain both actor and ref models always in VRAM? Or does it offload/reload only one at a given time?

[vadimkantorov]

Also at https://github.com/openpipe/deductive-reasoning they mentioned they tune Qwen2.5-32B with only 4 GPUs. Hmm

[pan-x-c]

How many 8-GPU nodes did you have to use to train a 32B model on long contexts?

We have tested Qwen2.5 32B GRPO training with 32k context on 48 A100-80G GPUs, 16 for Trainer and 32 for Explorer.