How to Optimize Distributed Training for Large Models
Optimize distributed training: choose parallelism strategy, tune batch size, minimize communication overhead, profile performance.
Choose the right parallelism strategy
Data parallelism (each GPU processes different batches), tensor parallelism (split model layers across GPUs), pipeline parallelism (split model into stages). For 70B+ models: use 3D parallelism (all three combined). Use Megatron-LM or DeepSpeed.
Tune your batch size
Global batch size = per-GPU batch size × number of GPUs × gradient accumulation steps. Target: 4M tokens for LLM training. Increase per-GPU batch until GPU memory is 90% full. Use gradient accumulation for effective larger batch.
Minimize communication overhead
Overlap computation and communication (compute next layer while communicating previous gradient). Use gradient bucketing (group small gradients). Choose NCCL for GPU-to-GPU communication. Use InfiniBand (400G) over Ethernet for multi-node.
Use mixed precision
Enable FP16 or BF16 for forward/backward passes (2x speedup, 50% memory). Keep FP32 master weights for accuracy. On H100, enable FP8 with Transformer Engine for 4x speedup. Mixed precision is the single biggest optimization.
Implement gradient checkpointing
Trade compute for memory: recompute activations during backprop instead of storing. Enables 3-5x larger models on same GPU. 20-30% slower but allows using fewer GPUs. Use when memory is the bottleneck.
Optimize data loading
Use num_workers > 0 in DataLoader. Pre-fetch with pin_memory=True. Use fast storage (NVMe). Pre-process data offline. Use webdataset for large datasets. Goal: GPU utilization > 80% (not waiting for data).
Profile your training
Use PyTorch Profiler to identify bottlenecks: (1) GPU compute time, (2) Communication time, (3) Data loading time, (4) CPU overhead. Optimize the bottleneck. Re-profile after each optimization. Common issues: data loading, small batch sizes, frequent synchronization.
Monitor and tune
Monitor: GPU utilization (>80% target), memory usage (<90%), communication overhead (<30% of total time), training throughput (samples/sec). Use Weights & Biases or MLflow for experiment tracking. Iterate on optimizations based on data.