How 10,000 GPUs train a single model — and why it explains the NVLink moat, the InfiniBand battle, and memory hierarchy
📍 Training loop · Memory pressure · DP / TP / PP · ZeRO · NVLink thesis~30 min
The Central Problem
Why training a large model requires thousands of GPUs working in concert
In Lesson 11 we established that training Llama 3 405B required ~38,000 H100s × 30 days. But how does a single model — one set of weights — get trained across thousands of physically separate GPUs? The answer determines why NVLink bandwidth commands a premium, why InfiniBand and Ethernet fight over different layers of the network, and why GPU memory capacity (not just bandwidth) is a binding constraint.
Core Thesis
Large model training requires three orthogonal types of parallelism — data, tensor, and pipeline — each with distinct communication patterns and bandwidth requirements. Tensor parallelism operates inside a layer and requires microsecond all-reduce over NVLink. Pipeline parallelism splits layers across node groups and tolerates millisecond latencies. Data parallelism replicates across cluster-scale all-reduce and drives InfiniBand vs. Ethernet economics. The 3D combination is why a single H100 training cluster looks nothing like a single-server system.
Section 1
The single-GPU training loop: what happens in one step
Before distributing training, understand what a single step requires. For each batch of tokens:
Step 1
Forward pass
Input tokens → model → output logits. Compute activations at every layer. Store them for backprop.
~1N FLOPs/token
Step 2
Loss
Compare logits to target tokens. Cross-entropy loss: scalar that measures prediction quality.
~tiny
Step 3
Backward pass
Backpropagation through every layer. Compute ∂L/∂W for each weight matrix. Requires stored activations.
~2N FLOPs/token
Step 4
Optimizer step
AdamW: update each weight using gradient + first moment m₁ + second moment m₂. Numerically stable in FP32.
~3N FLOPs/token
Step 5
Zero grad
Clear gradient buffers before next step. (Accumulate multiple micro-batches before clearing for gradient accumulation.)
~tiny
This is where C ≈ 6ND comes from: forward (~1N) + backward gradients w.r.t. weights (~2N) + backward gradients w.r.t. activations (~1N) ≈ 4N FLOPs/token for the backprop, plus the forward pass = 6N total. The optimizer step is typically amortized separately as it does not require GEMM operations on activations.
Mixed precision: forward and backward passes run in BF16 (fast, uses Tensor Cores). Optimizer states are stored in FP32 (necessary for numerical stability during weight updates — small gradients get flushed to zero in BF16). This is the "mixed precision" training standard introduced by NVIDIA.
Section 2
Why a single GPU can't train a large model: the memory math
Training a model requires far more memory than inference, because you must store gradients and optimizer states in addition to weights. For a model with N parameters trained with AdamW in mixed precision:
1,120 GB — doesn't fit in 7× H100s (7 × 80 GB = 560 GB)
1,120 GB
+ Activations (est.)
~300–500 GB
batch-dependent
Training Llama 3 70B requires approximately 1,400–1,600 GB of GPU memory — roughly 18–20 H100s just to hold the state, before factoring in activations and the actual batch data. This is the physical reason distributed training exists: no single GPU (or even a single 8-GPU node with 640 GB) can hold the full training state.
Section 3
The three types of parallelism — three different problems
Type 1
Data Parallelism (DP)
What it does: each GPU holds a complete copy of the full model. Different GPUs process different mini-batches. After each backward pass, all GPUs synchronize gradients via all-reduce and update weights identically.
Limitation: each GPU still needs to hold the full model — doesn't solve the memory problem for 70B+ models.
Communication: all-reduce of full gradient tensor after each backward pass. O(2 × model_size) bytes. Bandwidth sensitive, not latency sensitive.
Type 2
Tensor Parallelism (TP)
What it does: splits individual weight matrices within a single layer across GPUs. For a matrix multiply Y = XW, each GPU holds a column-slice of W and computes partial results; all-reduce combines them at each layer.
Limitation: requires all-reduce at every single layer — dozens per transformer block. Requires NVLink bandwidth (900 GB/s); InfiniBand (~25 GB/s) is ~36× too slow.
Communication: all-reduce after each attention + MLP block — very high frequency, latency-critical. Must be within a NVLink island.
Type 3
Pipeline Parallelism (PP)
What it does: assigns consecutive transformer layers to different GPU groups (stages). Stage 1 processes layers 1–10, passes activations to Stage 2 (layers 11–20), etc. Each stage holds only its slice of layers.
Limitation: pipeline "bubble" — each stage idles while adjacent stage is computing. Minimized using micro-batching (1F1B schedule). Latency-sensitive, not bandwidth-sensitive.
Communication: activation tensors passed between adjacent pipeline stages (point-to-point). Moderate bandwidth, high latency sensitivity. Can cross InfiniBand nodes.
3D Parallelism: How Megatron-LM organises a 1,024-GPU Cluster
3D parallelism combines all three strategies. Tensor parallelism (blue) operates within an 8-GPU NVLink island — splits each weight matrix. Pipeline parallelism (green) chains TP islands in sequence — passes activations forward, gradients backward over InfiniBand. Data parallelism (purple) runs identical pipelines on different mini-batches — gradient all-reduce at cluster scale. Each dimension addresses a different constraint: TP solves per-layer memory, PP solves total model depth, DP scales throughput.
Section 4
ZeRO: eliminating redundant memory with optimizer state sharding
In standard data parallelism, every GPU holds a full copy of model weights, gradients, and optimizer states. With 64 DP replicas training GPT-3, every byte of optimizer state is replicated 64 times — 64 copies of 2,100 GB = 134 TB of redundant memory. ZeRO (Zero Redundancy Optimizer), developed by Microsoft DeepSpeed, eliminates this redundancy.
Baseline DP
Full model × N_dp replicas Weights ✓ Grads ✓ Opt ✓
1× memory reduction
ZeRO-1
Shard optimizer states across DP Weights ✓ Grads ✓ Opt 1/N
With ZeRO-3 across 64 data-parallel workers: each GPU holds only 1/64 of the model parameters' optimizer state. A 70B model's 1,120 GB optimizer+gradient state becomes ~17.5 GB per GPU — fitting easily in 80 GB HBM. The tradeoff: ZeRO-3 requires gather/scatter operations before each layer (to collect the full weight matrix from across DP workers for each forward pass), adding communication overhead.
Gradient checkpointing (also called activation recomputation) is the complementary technique for activation memory: instead of storing all activations from the forward pass, store only checkpoint activations at regular intervals and recompute the intermediate ones during backward pass. This trades ~33% extra FLOPs for a significant reduction in activation memory.
Section 5
Communication bottlenecks: why each parallelism type needs different hardware
900 GB/s
NVLink 4.0
within NVL72 rack (H100)
400 Gbps
InfiniBand NDR
across nodes (~50 GB/s)
400 Gbps
400GbE (Ethernet)
across nodes (~50 GB/s)
18×
NVLink ÷ IB
900 / 50 = 18× bandwidth gap
The 18× bandwidth gap between NVLink and InfiniBand is not an accident — it's the constraint that shapes the entire 3D parallelism topology:
Tensor parallelism needs all-reduce at every layer (~50–100 all-reduces per forward pass). At 18 GB of activations per all-reduce for a 70B model on 8 GPUs, this requires ~900 GB/s throughput. NVLink delivers this; InfiniBand (50 GB/s) would make each step take 1,000× longer. TP must stay within the NVLink island.
Pipeline parallelism passes activation tensors between stages — typically 2–4 GB per micro-batch per stage boundary. At 50 GB/s InfiniBand, a 4 GB activation transfer takes ~80ms — tolerable across pipeline bubbles. PP crosses InfiniBand/Ethernet nodes.
Data parallelism all-reduce transmits gradient tensors (140 GB for Llama 3 70B in BF16) once per step. At 50 GB/s InfiniBand in a ring all-reduce, this takes ~2.8 seconds — significant but amortized over hundreds of micro-batches per step. With ZeRO-3, the gradient shards per-GPU reduce by N_dp, cutting the per-step communication proportionally. DP uses cluster-scale InfiniBand or Ethernet.
Why InfiniBand and Ethernet compete for different tiers
Tensor parallelism never crosses to InfiniBand — it's NVLink only. The contested ground is pipeline + data parallelism at cluster scale: InfiniBand offers RDMA (Remote Direct Memory Access) with ~2µs latency, which helps with pipeline's latency-sensitive micro-batch passing. Ethernet + RoCE (RDMA over Converged Ethernet) approaches parity for the bandwidth-sensitive data parallelism all-reduce. This is why the Ultra Ethernet Consortium (Broadcom + hyperscalers) is technically viable for the DP/PP tiers, but cannot replace NVLink for TP.
Section 6
NVL72: NVIDIA's answer to the TP island bottleneck
Standard HGX H100 servers connect 8 GPUs with NVLink per server. Tensor parallelism is therefore limited to TP=8. Scaling a model's TP dimension requires either accepting a cross-node NVLink hop (with latency and bandwidth degradation) or a new architecture.
NVIDIA's NVL72 (the DGX GB200 NVL72 rack) places 72 Blackwell GPUs in a single rack connected by fourth-generation NVLink via NVSwitch chips — allowing any GPU to communicate with any other GPU in the rack at full NVLink bandwidth. This makes the entire 72-GPU rack a single TP island.
The NVL72 Investment Thesis
NVL72 is not just a larger server — it changes what's computationally feasible. A 72-GPU TP island can tensor-parallelize a model 9× more aggressively than an 8-GPU server, allowing a single transformer layer's weight matrix to be split across 72 GPUs. For very large models or very latency-sensitive inference (where TP reduces per-token latency), NVL72 is architecturally superior to any alternative at any price. This creates a per-rack premium of ~$10M that customers pay specifically for the NVSwitch-connected topology — not just for raw TFLOPS.
Section 7 · Investment Lens
Investment implications of distributed training architecture
NVLink as structural moat
Tensor parallelism requires 900 GB/s all-reduce inside the TP island. This is a physics requirement of the training algorithm, not a preference. AMD's Infinity Fabric and custom ASICs can match NVLink bandwidth within a node — but NVIDIA's NVSwitch scales this to 72 GPUs in a rack with zero architecture changes. No competitor has an equivalent shipping product.
InfiniBand vs. Ethernet: the correct framing
The DP/PP tier (pipeline and data parallelism) is where InfiniBand (NVIDIA Mellanox) and Ethernet (Broadcom) compete. Ethernet + RoCE achieves comparable bandwidth for the all-reduce; InfiniBand has lower latency for pipeline's micro-batch passing. For pure training throughput (bandwidth-bound DP), the two are near-equivalent cost-adjusted — this is why hyperscalers can choose Ethernet without significant throughput loss.
Memory drives HBM demand
The 16 bytes/param training memory formula is the direct demand driver for HBM. As models scale — GPT-3 at 175B, GPT-4 estimated at 1T+, hypothetical GPT-5 at 5T+ — optimizer state memory grows linearly with parameters. Even ZeRO-3 reduces but doesn't eliminate the HBM demand (each GPU must gather weights before each layer). More params = more HBM per training cluster.
Training clusters vs. inference clusters
Training clusters prioritize NVLink bandwidth (TP island size) and all-reduce bandwidth (DP scale). Inference clusters prioritize HBM bandwidth (memory-bound decode) and NVLink for disaggregated prefill/decode routing. These are different optimal hardware configurations — training favors NVL72 rack topology; inference favors high-HBM-bandwidth GPU dense packing. As inference scales past training in total compute demand, the optimal hardware mix shifts.
Signals to watch
NVL72 / GB200 rack order velocity: each NVL72 rack is a signal that a customer is running TP-heavy workloads (large model training or latency-sensitive inference). Rack-scale NVLink purchases signal preference for NVIDIA's closed NVLink over commodity Ethernet for the high-bandwidth tier.
ZeRO-3 vs. TP scaling choice: when labs announce new model training strategies (e.g., "we shifted to larger TP degree"), this signals an increase in per-node NVLink bandwidth demand and potential NVL72 adoption. When labs scale DP with ZeRO, they signal preference for cheap scale-out networking.
Ultra Ethernet Consortium adoption: if a hyperscaler announces a full-scale training cluster on Ultra Ethernet (not InfiniBand), this signals RDMA over Ethernet is meeting the pipeline parallelism latency bar — potentially displacing Mellanox InfiniBand for the DP/PP tiers.
Model architecture choices: Mixture-of-Experts (MoE) models like GPT-4 and Mixtral use expert parallelism, a fourth parallelism dimension. MoE expert routing requires all-to-all communication (not just all-reduce) — different bandwidth pattern, potentially higher cluster-level bandwidth sensitivity.
Primary Sources
Recommended reading
Megatron-LM: Training Multi-Billion Parameter Language Models (Shoeybi et al., NVIDIA, 2019) — The foundational paper on tensor parallelism for transformers. Defines the column/row weight split for attention and MLP layers. Required reading for understanding TP topology.
1. Why does AdamW mixed-precision training require approximately 16 bytes per parameter — rather than the 2 bytes (BF16) used at inference?
The backward pass computes second-order gradients (Hessian diagonal) that must be stored alongside first-order gradients, requiring 8× additional memory for the curvature information Adam uses to scale per-parameter learning rates
Adam stores three additional FP32 tensors per parameter alongside the BF16 weights: an FP32 master weight copy (for numerical stability), a first moment (m₁), and a second moment (m₂) — adding 12 FP32 bytes to the 2 BF16 bytes already needed for forward pass weights
During training, activations from every transformer layer must be stored in full precision for backpropagation, and a 96-layer transformer with 16 GB activations per layer accounts for the majority of the 16-byte-per-parameter overhead
Mixed precision training uses BF16 for weights but FP64 for gradient accumulation to avoid catastrophic cancellation in the backward pass, and FP64 gradients are 4× larger than BF16 weights plus a matching FP32 copy for the optimizer update
2. Tensor parallelism requires an all-reduce operation at every transformer layer. Why does this mandate NVLink bandwidth rather than allowing InfiniBand?
InfiniBand uses a different RDMA protocol that is incompatible with the collective communication library (NCCL) used for tensor-parallel all-reduce operations in CUDA, requiring hardware-level protocol translation that adds unacceptable latency
Tensor parallelism operates on the activations within a single transformer layer, which are typically in FP8 precision for H100 Tensor Cores, and InfiniBand's line coding is incompatible with FP8 data transmission without precision-lossy transcoding
A 70B model on 8 GPUs generates roughly 18 GB of activation data per layer requiring all-reduce; with 80+ layers executing dozens of per-step all-reduces, the aggregate bandwidth demand exceeds 900 GB/s — NVLink's budget. InfiniBand at ~50 GB/s would slow each training step by 18×, negating the compute throughput gained from tensor parallelism
Pipeline parallelism already saturates InfiniBand bandwidth with its activation passing between pipeline stages, leaving no bandwidth headroom for tensor parallelism all-reduce on the same network, so tensor parallelism must use a separate physical interconnect
3. What problem does ZeRO-3 solve that tensor parallelism and pipeline parallelism do not address?
ZeRO-3 eliminates the pipeline bubble in pipeline parallelism by pre-fetching the next stage's activations while the current stage is executing its backward pass, overlapping compute and communication to achieve near-100% GPU utilization
TP and PP distribute model layers and weight matrices across GPUs, but every data-parallel replica still holds a full copy of optimizer states — ZeRO-3 shards optimizer states, gradients, and weights across data-parallel workers, eliminating redundant copies and enabling models far larger than any single GPU's memory to train with pure data parallelism
ZeRO-3 reduces the memory required for activations by recomputing them during the backward pass rather than storing them after the forward pass, freeing the HBM that TP and PP allocate for residual stream activations at each layer boundary
ZeRO-3 resolves numerical instability in BF16 gradient accumulation by fusing the all-reduce and cast operations into a single CUDA kernel, preventing precision loss that TP and PP training encountered when gradients were accumulated across multiple nodes before the FP32 master weight update
4. A hyperscaler announces they will build their next 16,000-GPU training cluster using 400 Gigabit Ethernet instead of InfiniBand. Which parallelism tier does this most directly affect, and is it architecturally problematic?
It affects tensor parallelism most directly, and it is fatally problematic: the all-reduce at every transformer layer requires sustained 900 GB/s which no Ethernet implementation achieves, so training throughput would drop by 18× compared to NVLink-based clusters
It affects data and pipeline parallelism across nodes, and it is largely viable: DP all-reduce is bandwidth-sensitive but tolerates millisecond latency, and Ethernet with RoCE/RDMA achieves comparable throughput to InfiniBand at equivalent link speeds — TP still runs on NVLink within each node, unchanged
It affects pipeline parallelism most directly and is moderately problematic: pipeline micro-batch passing is latency-sensitive (sub-millisecond), and Ethernet's 10–50× higher latency vs. InfiniBand increases pipeline bubble time, reducing overall cluster throughput by 30–40%
It has no architectural impact because modern training frameworks (Megatron-LM, DeepSpeed) abstract all network communication through NCCL, which automatically routes all-reduce operations through whichever physical fabric delivers the highest measured bandwidth at cluster initialization
5. NVIDIA's NVL72 rack places 72 GPUs on a shared NVLink fabric. What is the primary architectural advantage this provides for model training compared to eight standard 8-GPU HGX servers?
NVL72 allows all 72 GPUs to share a single unified memory address space, eliminating the need for explicit data movement between GPUs and allowing training frameworks to treat the entire rack as one logical GPU with 72× the HBM capacity of a single chip
NVL72 increases the per-GPU HBM bandwidth from 3.35 TB/s to 24 TB/s by allowing each GPU to directly read from any other GPU's HBM stack without transferring data over the NVSwitch — effectively pooling all 72 HBM stacks into one 192×80 GB = 15.4 TB memory system
NVL72 expands the tensor parallelism island from 8 GPUs (standard HGX server NVLink limit) to 72 GPUs, allowing model weight matrices to be split 9× more aggressively — enabling either larger models with the same per-GPU memory or lower per-token latency by parallelizing individual layer computation across a much wider GPU pool
NVL72 eliminates pipeline parallelism overhead by providing enough aggregate memory (72 × 80 GB = 5.76 TB) to fit a 400B+ parameter model entirely within the rack's HBM without distributing layers across pipeline stages, removing all inter-rack activation passing latency
Questions worth exploring: "How does Mixture-of-Experts (MoE) parallelism differ from TP/PP/DP?" · "What is sequence parallelism and when is it used?" · "How does Flash Attention reduce activation memory in the backward pass?" · "Why do large training runs sometimes fail midway and how does checkpointing help?" · "What does NVIDIA's NVLink Switch ASIC actually do inside NVL72?" Ask your teacher any of these to go deeper.
Coming Up — Lesson 14
Post-training alignment — RLHF, DPO, and GRPO: pretraining produces a next-token predictor. Post-training alignment turns it into a useful assistant. We'll cover Reinforcement Learning from Human Feedback (RLHF), Direct Preference Optimization (DPO), and the newer Group Relative Policy Optimization (GRPO) used in DeepSeek-R1 — and why the compute intensity of alignment is driving a new category of demand distinct from pretraining.