In Lesson 15 you learned NVIDIA's moat has three layers — NVSwitch topology, the CUDA ecosystem, and TSMC packaging — and that the topology layer is engineering, not a law of nature. This lesson stress-tests that. AMD has matched, even beaten, NVIDIA on raw silicon specs for two years. So why is NVIDIA still ~90% of the datacenter GPU market? The answer tells you exactly what the moat is — and what would breach it.
Core thesis: AMD has reached chip-level parity (and leads on HBM capacity), so the contest is no longer about the GPU die — it's about the two things outside the die: rack-scale interconnect and the software stack. AMD's wedge is inference, where memory capacity matters more than rack-scale topology and where the CUDA moat is thinnest. The bull case is not "AMD beats NVIDIA" — it's "AMD becomes a real second source in inference, funded by hyperscalers who refuse to depend on one vendor."
01 — The Silicon Is No Longer the Gap
Spec Parity, Capacity Lead
Read this table the way you learned in L15: ask what workload each row serves. The striking pattern — AMD wins on HBM capacity generation after generation.
| Spec |
NVIDIA H100 |
AMD MI300X |
NVIDIA B200 |
AMD MI350X / MI355X |
| HBM capacity | 80 GB | 192 GB | 192 GB | 288 GB |
| HBM bandwidth | 3.35 TB/s | 5.3 TB/s | 8.0 TB/s | ~8.0 TB/s |
| FP16/BF16 | 989 TF | ~1,300 TF | 4,500 TF | competitive |
| FP8 / FP4 | FP8 | FP8 | FP8 + FP4 | adds FP6 + FP4 |
| Process | TSMC N4 | TSMC N5+N6 (chiplet) | TSMC N4P (dual-die) | TSMC N3 |
| Scale-up topology | NVL8 (900 GB/s) | 8-GPU node (Infinity Fabric) | NVL72 (1.8 TB/s, TP=72) | 8-GPU node (Helios rack = 2026) |
Why the capacity lead matters. Recall the KV-cache arithmetic from L15: a big model + long context is a memory-capacity problem. A model that fits in one MI300X's 192 GB (or one MI350X's 288 GB) needs fewer GPUs to serve than on an 80 GB H100 — fewer GPUs means lower cost and no cross-GPU tensor-parallel overhead. AMD chose to compete exactly where its capacity advantage converts directly into inference TCO.
So if the silicon is at parity, why ~90% NVIDIA share? Because buyers don't buy a die — they buy a system you can deploy in a week and a software stack your engineers already know. That's the whole ballgame, and it lives in the next two sections.
02 — The Three-Layer Moat, Re-examined
Where It Holds, Where It Erodes
Layer 1 — Rack-scale interconnect (NVSwitch)
Holds — for training
From L13: tensor parallelism all-reduces after every layer, demanding ~900 GB/s+ — only NVLink delivers it. NVIDIA's NVSwitch scales the TP island to 72 GPUs (NVL72) in one rack. AMD's Infinity Fabric tops out at ~8 GPUs per node; beyond that, traffic falls to Ethernet/InfiniBand (~50 GB/s) — far too slow for TP. For giant training runs, this is a genuine, physics-rooted gap. The breach attempt: AMD's announced Helios rack (MI400 series, 2026) plus the open UALink standard aim to deliver a rack-scale, ~72-GPU scale-up fabric. Not shipping yet — but it's the exact product that would close this layer.
Layer 2 — Software ecosystem (CUDA vs ROCm)
Eroding — fastest at inference
CUDA is 15+ years of libraries (cuBLAS, cuDNN, NCCL) and hand-tuned kernels; AMD's ROCm/HIP runs most PyTorch models but trails on last-mile kernel tuning (the final ~15–20% of utilization), and historically on stability. Why it's eroding: most inference no longer goes through hand-written CUDA — it goes through abstraction layers (PyTorch + torch.compile/Triton, vLLM, SGLang) that target multiple backends. Each model that runs well on ROCm via these layers removes a brick from the wall. Training still leans on CUDA's deep, tuned stack.
Layer 3 — TSMC advanced packaging (CoWoS / HBM)
Shared — not a moat vs AMD
Both NVIDIA and AMD buy HBM from SK Hynix/Micron/Samsung and CoWoS capacity from TSMC. This is a
shared bottleneck, not an advantage for either — it constrains both. AMD even leans into it, packing more HBM per package. (See
L17 and
L12.)
03 — Why Inference Is AMD's Wedge
The Battle Splits in Two
The single most important framing in this lesson: training and inference are different markets with different moats.
| Dimension | Large-scale training | Inference (esp. decode / serving) |
|---|
| Binding constraint | Rack-scale TP bandwidth + MFU | HBM capacity + bandwidth per GPU |
| Needs NVL72 topology? | Yes — the moat bites hard | Often no — fits in fewer GPUs |
| Software intensity | Deep CUDA/NCCL tuning | Mostly PyTorch/vLLM abstractions |
| AMD competitiveness | Weak (topology + ROCm gap) | Strong (capacity lead, thin CUDA moat) |
| Demand trajectory | Large, lumpy | Growing faster, more recurring |
This is why AMD's real-world wins are in inference: Meta serves Llama models on MI300X, Microsoft Azure and Oracle offer MI300X instances, and AMD's data-center GPU revenue went from ~zero to ~$5B in 2024 on this wedge. As inference overtakes training in total compute (a thread running through L8, L14, L15), the part of the market where AMD competes is the part that's growing fastest.
04 — The Industry's Collective Assault on the Moat
UALink and Ultra Ethernet
AMD isn't fighting alone. The hyperscalers — who hate single-vendor dependency and the margins it funds — are backing two open standards aimed squarely at NVIDIA's two interconnect moats:
- UALink — an open scale-up (GPU-to-GPU, rack-scale) standard, the consortium answer to NVLink/NVSwitch. Backers include AMD, Broadcom, Google, Intel, Microsoft, Meta. This is the collective attempt to commoditize Layer 1.
- Ultra Ethernet (UEC) — an open scale-out (cluster) standard to make Ethernet+RoCE a full peer to InfiniBand (recall L13: Meta already proved RoCE works at 100K+ GPUs). Broadcom is the prime beneficiary here.
If these mature and get adopted, NVIDIA's interconnect advantage becomes a standards-body feature anyone can buy — and the contest collapses back to silicon + software, where AMD is far more competitive. This is the structural reason the bear case on NVIDIA's durability is credible.
05 — Investment Implications
The Second Source, Priced
AMD is a call option on inference + CUDA erosion
You're not betting AMD beats NVIDIA. You're betting it captures a meaningful slice of the fastest-growing (inference) segment as a funded second source. Even 10–20% of datacenter accelerator share would multiply AMD's data-center GPU revenue off the ~$5B 2024 base. The capacity lead + ROCm-via-PyTorch is the mechanism.
Hyperscaler dual-sourcing is structural demand
Buyers want a credible #2 to negotiate NVIDIA pricing and de-risk supply. That gives AMD a floor of demand almost regardless of a pure perf/$ win — the same logic that funds custom ASICs (L10). Watch named design wins and committed capacity at MSFT/Meta/Oracle/OpenAI.
The NVIDIA mirror
Every brick AMD removes from the CUDA/NVLink wall is a brick out of NVIDIA's durability (not its current dominance). NVDA's near-term moat is intact; the question THESIS.md tracks is how fast Layers 1–2 erode. AMD progress and NVDA durability are two readings of the same gauge.
2026 is the verdict year
AMD's Helios MI400 rack + UALink is the product that would close the topology layer. Its qualification and hyperscaler adoption (vs. NVIDIA's Rubin generation) is the single most important catalyst to track. Until it ships and is qualified at scale, the training moat holds.
Thesis Update
AMD & NVDA in THESIS.md
AMD — promote to grounded (★★). Key technical assumption: inference (memory-bound, often single-node) is a large, fast-growing pool where the topology moat matters less and AMD competes on HBM capacity + price, with ROCm reaching "good enough" via PyTorch/Triton abstractions.
- Bull confirmed if: AMD ships a qualified, rack-scale Helios/UALink fabric adopted by a major hyperscaler and ROCm reaches near-parity in production serving.
- Bull broken if: AMD fails to gain material inference share despite competitive specs — proving the software/topology moat holds regardless of silicon.
- For NVDA: the falsifier of "moat durability" is the same event read from the other side — a qualified rack-scale NVSwitch competitor + ROCm parity + UALink/UEC adoption.
Primary Source
Go Deeper
Read first: SemiAnalysis — "MI300X vs H100 vs H200" benchmark deep-dive — the highest-signal independent analysis of where AMD's silicon genuinely competes and where the ROCm software gap shows up in real benchmarks. (In RESOURCES.md.)
Also: AMD's annual Advancing AI keynote (MI350/MI400 + Helios + ROCm roadmap) for the company's own framing — read it skeptically against the SemiAnalysis benchmarks.
Comprehension Check
Quiz — 5 Questions
Select the best answer for each.
1. AMD MI300X (192 GB) and MI350X (288 GB) lead NVIDIA on HBM capacity. Which workload does that advantage most directly help?
Giant pretraining runs that span thousands of GPUs
Serving a large / long-context model in as few GPUs as possible
All-reduce communication in tensor parallelism
Reducing the CUDA-to-ROCm porting effort
2. Despite spec parity, NVIDIA holds ~90% share on large-scale training. The most defensible reason is:
AMD chips cannot run BF16 precision
NVSwitch scales tensor parallelism to 72 GPUs; AMD Infinity Fabric caps at ~8 per node
AMD cannot buy HBM from SK Hynix
TSMC refuses to make AMD's training GPUs
3. Why is the CUDA moat thinner for inference than for training?
Inference doesn't use the GPU's tensor cores at all
Most inference runs through PyTorch/vLLM/Triton abstractions that target multiple backends
Inference is done on CPUs, so the GPU vendor doesn't matter
ROCm only supports inference and cannot do training at all
4. What are UALink and Ultra Ethernet, in the moat story?
NVIDIA's two newest proprietary interconnects
Open standards (scale-up and scale-out) backed by hyperscalers to commoditize NVIDIA's interconnect moats
Two AMD-only GPU models due in 2026
Memory standards that replace HBM
5. Which single event would most strongly confirm the AMD bull case (and falsify NVIDIA's "durable topology moat")?
AMD releasing a GPU with more HBM than NVIDIA again
A qualified, rack-scale Helios/UALink fabric adopted at volume by a major hyperscaler
AMD's stock outperforming NVIDIA for one quarter
AMD cutting MI350X prices below the H100
From your instructor: The frame to keep — the GPU die is at parity; the moat lives outside the die, in rack-scale interconnect and software, and it's thinnest where the market is growing fastest (inference). AMD is the funded second source, not the giant-killer. Ask me anything: the UALink-vs-NVLink mechanics, how to read SemiAnalysis benchmarks past the marketing, or how AMD's Helios stacks up against NVIDIA's Rubin roadmap.