NVIDIA Blackwell: B200 Architecture

You now have the full stack: scaling laws tell you how much compute is needed, distributed training explains how it's done, HBM explains the memory bottleneck, CoWoS explains the packaging, and alignment explains what happens after training. With that foundation, we can evaluate Blackwell not as a marketing spec sheet but as an engineering response to specific physics constraints.

Every B200 design decision traces to something you already understand. The goal of this lesson is to close that loop.

Core thesis: The B200's architecture is a systems-level response to the arithmetic intensity gap between prefill and decode. The NVL72 topology, 8 TB/s HBM3e, FP4/FP8 precision hierarchy, and disaggregated serving are four coordinated answers to the same problem: LLM inference is memory-bandwidth-bound, but heterogeneous (prefill isn't). NVIDIA's hardware moat is the NVSwitch topology that enables TP=72 and KV-cache migration within a single rack — something no competitor offers today.

What Changed and Why

Specs only mean something when you know what problem they're solving. Read each row asking: what workload does this address?

Key observation: H200 vs H100 is almost purely a memory upgrade — same die, same compute, more HBM3e stacks. This was NVIDIA shipping a pure bandwidth answer before Blackwell was ready. B200 jumps both compute (4.5×) and bandwidth (2.4×). The compute gain is larger because NVIDIA added FP4 and doubled Tensor Core throughput — mostly a prefill/training benefit. The bandwidth gain compounds the decode improvement H200 started.

FP32 → BF16 → FP8 → FP4

Each precision halving doubles peak TFLOPS and halves memory footprint — but at the cost of dynamic range. The engineering question is: can the model tolerate it?

Why FP4 is now viable for inference: BF16 has 8-bit mantissa; FP8 E4M3 has 3-bit mantissa; FP4 E2M1 has only 1-bit mantissa. For the longest time, 4-bit weights caused accuracy collapse on sensitive activations. What changed:

• Post-training quantization (PTQ) matured. Techniques like GPTQ, AWQ, and SmoothQuant smooth activation outliers before quantizing, preserving accuracy at INT4/FP4.
• Micro-scaling (MXFP4). NVIDIA's B200 uses microscaled FP4 — small groups of values (16–32 elements) share a scaling factor. This recovers most of the dynamic range lost from the narrow exponent without storing full FP8.
• Inference-only use case. FP4 is for inference weights, not training. The quality bar is measured by benchmark degradation (<1% on most evals), not gradient stability.

At FP4, a 70B model's weights compress to ~35 GB — well within B200's 192 GB HBM. The 18,000 TFLOPS peak moves the B200 well into compute-bound territory for batched prefill, enabling much higher throughput per dollar on prompt-heavy workloads.

From 8-GPU Nodes to 72-GPU Racks

Recall from Lesson 13: Tensor Parallelism (TP) requires all-reduce communication between GPUs after every single transformer layer, at ~900 GB/s NVLink bandwidth. This is why TP is confined to a NVLink island — it's physically impossible to do it at InfiniBand speeds.

What TP=72 actually changes:

• Elimination of pipeline parallelism for mid-sized models. Llama 3 70B at TP=72 fits on 72 B200s (each holds ~1B params). No pipeline stages → no pipeline bubble overhead (~15–30% of training time eliminated).
• KV cache can migrate within the rack without leaving NVLink. Critical for disaggregated serving (next section).
• Fewer inter-node communication events. When TP covers the entire model, you only need inter-rack communication for Data Parallelism gradient syncs — which tolerates InfiniBand latency.

Why LLM Serving Needs Two Different Hardware Modes

Recall from Lesson 8 (LLM Inference at Scale): prefill processes the entire input prompt in parallel (compute-bound, ~300 FLOPs/byte), while decode generates one token at a time (memory-bandwidth-bound, ~2 FLOPs/byte). These are so different they operate near opposite ends of the roofline model — and optimal hardware for one is suboptimal for the other.

Coupled mode (traditional): A single GPU pool handles both phases. Every decode step, the entire weight tensor is loaded from HBM even though only one token is being generated. Long prompts cause "time-to-first-token" (TTFT) spikes that hurt latency SLAs. Throughput-optimized batching conflicts with latency-optimized scheduling.

Disaggregated mode (Blackwell's target architecture):

After prefill completes, the KV cache is migrated from the prefill GPU to the decode GPU over NVLink (within the NVL72 rack). At 1.8 TB/s, transferring a 10GB KV cache takes ~5.5 ms — fast enough to not dominate latency.

Why NVL72 is the enabler: KV cache migration between prefill and decode pools requires high-bandwidth, low-latency communication. On a traditional cluster, this would traverse InfiniBand (~50 GB/s), adding hundreds of milliseconds per long-context request. On NVL72, the migration happens over NVLink within the same rack, an 18× bandwidth advantage.

Impact on throughput: Published benchmarks show disaggregated serving with NIM on GB200 NVL72 delivers roughly 3× higher token throughput vs. H100 for long-context inference (32K+ tokens), compared to the ~2× improvement you'd expect from raw bandwidth alone. The topology multiplier is real.

Why NVIDIA Went Multi-Die on Blackwell

The B200 is built from two dies connected by a 10 TB/s NVLink-C2C (chip-to-chip) interconnect. This is not a GPU chiplet design — both dies run as a unified memory space, exposed to software as a single GPU.

Why not one big die? The H100 die is ~814 mm² on TSMC N4. At Blackwell's transistor density, a single die with B200's compute would require ~1,600+ mm² — well beyond TSMC's reticle limit (~850 mm²) and would suffer catastrophic yield losses. Two smaller dies have ~2× the combined yield of one enormous die at the same defect density.

The NVLink-C2C interconnect: 10 TB/s is 11× faster than PCIe 5.0 x16 and 5× faster than NVLink between separate GPUs. From software, the two dies appear as unified HBM address space — the C2C bridge handles coherence transparently. This is what makes the dual-die design transparent to CUDA and existing frameworks.

AMD, Broadcom, and Google vs. NVL72

This is not a "NVIDIA has no competition" narrative — it's a first-principles analysis of what the moat actually is and where it can be breached.

The three-layer moat — and how each can be breached:

Layer 1: NVSwitch topology. AMD's InfinityFabric tops at 8 GPUs per node; scaling beyond requires InfiniBand, which can't sustain TP at ≥72 GPUs. To breach this, AMD would need a rack-scale NVSwitch equivalent — a separate product with massive R&D. AMD has announced no such product as of mid-2025. Breach likelihood: 2–3 years out at earliest.

Layer 2: Software ecosystem. CUDA has 15+ years of kernel optimizations, libraries (cuBLAS, cuSPARSE, NCCL), and frameworks tuned to its memory model. ROCm runs most PyTorch models but lacks the last-mile kernel tuning that captures the final 15–20% of GPU utilization. Breach mechanism: Meta, Microsoft, and Google are all investing in ROCm compatibility. Each percentage point of CUDA dependency removed weakens this layer.

Layer 3: CoWoS/HBM ecosystem. B200 requires TSMC CoWoS-L at maximum interposer scale. AMD also needs TSMC CoWoS for MI350X. This is a shared dependency — neither benefits from the other's exclusion here. The constraint is TSMC capacity, not any one vendor's IP. Breach likelihood: not applicable — this is a shared bottleneck.

Bear case on NVIDIA moat: If AMD achieves rack-scale InfinityFabric + ROCm reaches CUDA parity, the moat shrinks to software switching costs. Hyperscalers have shown willingness to invest in migrations (Google TPU, Meta's RSC). The NVLink topology advantage is real today but not permanent physics — it's an engineering gap, not a law of nature.

The Long-Context Memory Problem

Every B200 spec discussion fixates on the 8 TB/s bandwidth. But the 192 GB capacity matters equally for production inference. Here's the arithmetic:

KV cache per token (Llama 3 70B):

• Layers: 80
• KV heads: 8 (GQA — grouped query attention)
• Head dimension: 128
• Per token, per layer: 2 (K+V) × 8 heads × 128 dim × 2 bytes (FP16) = 4,096 bytes = 4 KB
• Per token, all layers: 4 KB × 80 = 320 KB
• At 128K context: 320 KB × 128K = 40 GB per sequence

A single 128K-context Llama 3 70B sequence uses 40 GB of KV cache. With FP4 weights at ~35 GB, a B200 with 192 GB HBM can hold: 192 − 35 = 157 GB for KV cache → 3–4 simultaneous 128K sequences. That's it.

Why this matters for the investment thesis: Context length is growing (128K → 1M+ tokens). Every 8× increase in context length multiplies KV cache 8×. The only solutions are: more HBM per GPU (HBM4), more GPUs per serving instance, or architectural changes (linear attention, state space models). This is why HBM4 (expected B300) — which doubles the bus width to 2,048 bits and targets ~3 TB/s per stack — is critical for long-context inference, not just bandwidth.

What Blackwell Means for the Value Chain

The second-order implication on power infrastructure: B200 at 1,000 W TDP × 72 GPUs = 72 kW per rack (GPU only). A full NVL72 rack with networking and cooling exceeds 120 kW. This is 3–4× the power density of a standard server rack. Hyperscalers are now building liquid-cooled facilities specifically for NVL72 — capital expenditure extending to cooling infrastructure, power delivery, and facility design. Vertiv, Eaton, and data center REITs are infrastructure beneficiaries.

Go Deeper

Recommended reading: NVIDIA Blackwell Architecture Technical Brief — NVIDIA's official architecture whitepaper. Covers the transformer engine, FP4 microscaling, NVLink C2C architecture, and NIM inference microservices. Dense but technically precise.

Also recommended: NVIDIA Blackwell Platform Overview (NVIDIA Developer Blog) — higher-level companion piece with system architecture diagrams.