Env: transformers 5.12.1, deepspeed 0.18.9, torch 2.12.0+cu130, peft 0.19.1, bitsandbytes 0.49.2, accelerate 1.14.0; 8x B200 (178GB) / 2TB RAM; model MiniMaxAI/MiniMax-M3 (428B sparse MoE VL, minimax_m3_vl).
If from_pretrained (with an active HfDeepSpeedConfig) runs before the torch.distributed process group is initialized, zero.Init sees world_size=1 and creates each parameter whole on every rank. With deepspeed --num_gpus 8, all 8 ranks then load the full 428B model onto their own GPU -> CUDA OOM (~170GB/GPU). No warning is emitted — it is indistinguishable from a "model too big" OOM. Adding deepspeed.init_distributed() before from_pretrained fixes it (each rank then holds ~1/8, ~94GB/GPU).
Ask: when a multi-GPU launcher env is present (RANK/WORLD_SIZE set) but the process group is uninitialized at zero.Init time, warn or error loudly (or auto-init). The silent single-rank fallback is very expensive to debug.
Env: transformers 5.12.1, deepspeed 0.18.9, torch 2.12.0+cu130, peft 0.19.1, bitsandbytes 0.49.2, accelerate 1.14.0; 8x B200 (178GB) / 2TB RAM; model MiniMaxAI/MiniMax-M3 (428B sparse MoE VL, minimax_m3_vl).
If
from_pretrained(with an activeHfDeepSpeedConfig) runs before the torch.distributed process group is initialized,zero.Initsees world_size=1 and creates each parameter whole on every rank. Withdeepspeed --num_gpus 8, all 8 ranks then load the full 428B model onto their own GPU -> CUDA OOM (~170GB/GPU). No warning is emitted — it is indistinguishable from a "model too big" OOM. Addingdeepspeed.init_distributed()beforefrom_pretrainedfixes it (each rank then holds ~1/8, ~94GB/GPU).Ask: when a multi-GPU launcher env is present (RANK/WORLD_SIZE set) but the process group is uninitialized at
zero.Inittime, warn or error loudly (or auto-init). The silent single-rank fallback is very expensive to debug.