Model Architectures · Advanced
NeMo and Multi-GPU Training: Parallelism, NCCL and Megatron Techniques
NeMo is NVIDIA's framework for customising and training models at scale, built on Megatron-style parallelism and NCCL collectives. Learn the four kinds of parallelism, when each is needed, and why most teams should fine-tune rather than pre-train.
- 2 hours lesson
- advanced level
- NVIDIA AI Track
- Narrated presentation
What you’ll learn
- Data, tensor, pipeline and sequence/expert parallelism, and how they combine
- Tensor parallelism belongs inside an NVLink domain; pipeline parallelism crosses nodes
- FSDP/ZeRO-style sharding of optimiser state, gradients and parameters
- NCCL collectives and diagnosing communication-bound scaling with topology tools
- NeMo recipes: pre-training, SFT, PEFT, alignment, evaluation and export to TensorRT-LLM/NIM
- The customisation ladder — stop at the cheapest rung that meets the requirement
Inside this lesson
A narrated, slide-by-slide presentation with 1 prediction exercise, 1 knowledge check, 3 videos. You answer as you go, and the platform checks your work.