No notes. Retrieve first, reveal second. A spaced, interleaved test of the whole stack — from sand to served token, and from one dollar of capex to where it lands.
How to use this — and why it works. This is not a lesson; it's a test, and the difficulty is the point. For each card below, answer out loud or in writing before you click reveal. The struggle to retrieve — even a partial or failed attempt — is what converts fragile fluency (it felt familiar when I read it) into durable storage strength (I can produce it cold weeks later). Reading the answer without trying first gives you the illusion of mastery and almost none of the retention. The cards are deliberately interleaved — they jump between manufacturing, model architecture, and economics — because mixing topics forces you to first decide which mental model applies, which is the harder, more transferable skill.
Part A — Free Recall (interleaved across all phases)
Twelve prompts, deliberately out of order. Produce the answer before revealing. If you can't, note it — that gap is your review list.
L4 · TSMCL17 preview
1. Name the three distinct things that make TSMC's leading-edge position so hard to replicate. Not "they're good" — the structural reasons.
Model answer
Process learning / yield as a flywheel: yield improves with cumulative volume. TSMC's lead in wafers run = a lead in defect-density tuning that money alone can't shortcut.
The ecosystem & trust moat: EDA flows, IP libraries, and ~500 customers co-design to TSMC's PDK. A pure-play foundry that never competes with its customers earns trust Samsung/Intel can't.
Capital + tool access: ~$30B+/yr capex and first access to ASML EUV/High-NA tools. The capital intensity itself is a barrier, and the tool supply is gated.
Investor lens: the moat is durable because all three compound; a challenger must match all of them at once.
L8 · InferenceL15 · Blackwell
2. Inference has two phases with opposite hardware appetites. Name them, state which is compute-bound vs. memory-bandwidth-bound, and give the rough arithmetic intensity (FLOPs/byte) of each.
Model answer
Prefill — processes the whole prompt in parallel → compute-bound, ~300 FLOPs/byte. Loves FP8/FP4 TFLOPS.
Decode — generates one token at a time, reloading all weights each step → memory-bandwidth-bound, ~2 FLOPs/byte. Loves HBM bandwidth.
This mismatch is exactly why Blackwell supports disaggregated serving (separate prefill and decode GPU pools) and why decode benefits from H200/B200's bandwidth jump.
L11 · Scaling laws
3. State the Chinchilla result in one sentence, and give the training-FLOPs rule of thumb (the "6ND" formula). What does each letter mean?
Model answerChinchilla: for a fixed compute budget, model size and training tokens should scale together — roughly ~20 tokens per parameter — earlier models (GPT-3) were badly under-trained (too big, too few tokens).
Training compute ≈ 6ND, where N = number of parameters, D = number of training tokens. The 6 = ~2 FLOPs/param forward + ~4 FLOPs/param backward, per token.
L13 · Training
4. In mixed-precision AdamW training, why does each parameter cost ~16 bytes of memory? Break down the bytes.
Model answer
BF16 weight — 2 bytes
BF16 gradient — 2 bytes
FP32 master weight copy — 4 bytes
FP32 Adam first moment (m₁) — 4 bytes
FP32 Adam second moment (m₂) — 4 bytes
= 16 bytes/param, before activations. So Llama 3 70B needs ~1.1 TB just for training state → ~18–20 H100s before you fit any activations. The FP32 copies exist because tiny gradients round to zero in BF16.
L13 · ParallelismL6 · CUDA moat
5. There are three orthogonal kinds of parallelism. Name them, say what each splits, and rank them by interconnect bandwidth requirement. Which one is the physics basis for NVIDIA's NVLink/NVSwitch moat?
Model answer
Tensor (TP) — splits weight matrices within a layer; all-reduce after every layer → needs ~900 GB/s+ → NVLink-only. This is the moat.
Pipeline (PP) — splits the model by depth (layer groups); passes activations stage-to-stage → latency-sensitive, tolerates InfiniBand.
Data (DP) — replicas on different batches; all-reduce gradients once per step → tolerates InfiniBand/Ethernet.
TP's per-layer all-reduce is a physics requirement for high bandwidth. NVSwitch extends the TP island from 8 GPUs (HGX) to 72 (NVL72) — the gap AMD has no shipping answer to.
L12 · HBML5 · Packaging
6. How does HBM physically reach multiple TB/s of bandwidth when a normal DRAM stick can't? And what production factor — not design — limits how much HBM can ship?
Model answerMechanism: bandwidth = bus width × clock. HBM stacks DRAM dies vertically, connected by TSVs (through-silicon vias), giving an enormous 1,024-bit bus per stack (vs. 64-bit for DDR), sitting right next to the GPU on a silicon interposer (CoWoS) so the wires are short. HBM4 doubles the bus to 2,048-bit.
Production limiter: yield. Stacking 8–12 known-good dies + TSVs means defects compound — and the CoWoS interposer that co-packages HBM with the GPU is itself capacity-constrained at TSMC. Yield, not design, gates supply.
L7 · Transformer
7. In one or two sentences, what does self-attention compute, and why is the KV cache the thing that dominates inference memory?
Model answer
Self-attention lets each token weight every other token via Query·Key similarity, then mixes their Values — that's how context is incorporated. During generation, the Keys and Values of all past tokens are cached (the KV cache) so they aren't recomputed each step. KV cache grows linearly with sequence length × layers × heads, so at long context it can exceed the model weights themselves (e.g. ~40 GB for one 128K-token Llama-70B sequence). GQA reduces it by sharing KV heads.
L10 · ASICL9 · TPUL6 · CUDA
8. A hyperscaler can buy NVIDIA GPUs or build a custom ASIC (via Broadcom/Marvell, or Google's TPU). Give the core trade-off — what the ASIC wins, and the two things the GPU keeps.
Model answerASIC wins: better perf-per-watt and lower $/token for a known, stable workload at huge volume — you strip out general-purpose silicon and NVIDIA's margin.
GPU keeps: (1) flexibility — ASICs are bets on an architecture that may be obsolete before the ~2-yr design cycle ends; (2) the CUDA software ecosystem — 15+ years of kernels/libraries an ASIC team must rebuild. Investor lens: ASICs make sense only for the few hyperscalers with workloads big and stable enough to amortize the NRE; Broadcom monetizes this as the merchant ASIC arms dealer.
L14 · Alignment
9. Trace the compression from RLHF→DPO→GRPO: how many models must be held in memory for each, and what did each step eliminate? Then the investor twist: why is "GRPO is 10× cheaper" mostly irrelevant to GPU demand?
DPO: 2 models — derives the reward analytically from the policy's log-probs, eliminating the reward model (and critic).
GRPO: 2–3 — replaces the critic with group statistics (advantage = (Rᵢ − mean)/std over G samples); with verifiable rewards (math/code), the reward model drops too → 2.
Investor twist: alignment is only ~1–5% of total training compute. A 10× efficiency gain there cuts <1% of total cost — immaterial to GPU demand. The real signal is capability access (smaller labs can now train reasoners) and, via reasoning models, more inference tokens per query → inference GPU demand.
L15 · Blackwell
10. Why is the B200 a dual-die design rather than one big chip? Tie it to a physics limit and a yield argument.
Model answer
A single die with B200's compute would need ~1,600 mm² — beyond TSMC's reticle limit (~850 mm²). Yield falls with area: two ~800 mm² dies have far better combined economics than one giant die, because defects scale with area and a single defect kills the whole die. The two dies are joined by NVLink-C2C at 10 TB/s and appear to software as one unified GPU. Constraint that actually gates shipment: the CoWoS-L interposer (~2,200 mm², 2 dies + 8 HBM stacks), made only by TSMC.
L2/L3 · China
11. Why has China — despite enormous capital — not produced a TSMC-equivalent at the leading edge? Name the single most decisive chokepoint and one structural reason capital alone fails here.
Model answerDecisive chokepoint: no EUV. ASML (Dutch) is the sole EUV maker and export controls block sales to China → SMIC is stuck pushing DUV multi-patterning, which is costly and yield-limited beyond ~7nm.
Why capital alone fails: leading-edge yield is accumulated process knowledge — thousands of tuned recipe steps learned over years of high-volume runs. You cannot buy that learning curve; subsidies poured into firms without it have produced low yields and notable failures (the "build fabs, hope for yield" trap).
L1 · Value chainsynthesis
12. From memory, list the semiconductor value chain in order — EDA → … → end customer — and name the dominant player and competitive structure (monopoly/duopoly/oligopoly) at each load-bearing node.
Model answer
EDA tools: Cadence + Synopsys — duopoly.
IP cores: Arm — near-monopoly in CPU ISA licensing.
HBM memory: SK Hynix > Samsung > Micron — oligopoly.
Advanced packaging (CoWoS): TSMC-dominated.
GPU/accelerator: NVIDIA dominant; AMD #2; ASICs via Broadcom — competitive but NVIDIA-led.
Cloud / end customer: hyperscalers (MSFT, Google, Amazon, Meta).
The durable pricing power sits where structure is monopoly/duopoly: ASML, TSMC, the EDA duo.
Part B — Interleaved Quiz (auto-graded)
Six questions, mixed topics
Each jumps to a different part of the stack. Choose the best answer.
1. A team upgrades from H100 (3.35 TB/s) to H200 (4.8 TB/s) with identical compute. Which workload improves most?
Pretraining a model from scratch on a fixed token budget
Low-batch decode serving of a deployed model
Large-batch FP8 prefill of long prompts
PPO fine-tuning that holds four models resident
2. Which parallelism strategy is the one that physically requires NVLink-class bandwidth and so defines NVIDIA's NVSwitch moat?
Data parallelism, because gradient all-reduce is the largest transfer
Pipeline parallelism, because activations cross stage boundaries
Tensor parallelism, because it all-reduces after every single layer
Expert parallelism, because MoE routing is bandwidth-heavy
3. Chinchilla's central finding about how to spend a fixed compute budget is best summarized as:
Make the model as large as memory allows, tokens matter little
Scale parameters and training tokens together, roughly 20 tokens per parameter
Train smaller models far longer until validation loss flattens
Compute scales with the square of parameter count regardless of data
4. Which factor most directly limits how much HBM the industry can ship in a given quarter?
The DDR5 spot price set by PC demand
Stacking/TSV yield and CoWoS interposer capacity
The availability of GDDR controllers from NVIDIA
Export licensing of EUV scanners to memory makers
5. The single most decisive reason China cannot yet field a leading-edge logic foundry is:
A shortage of capital to fund fab construction
No access to ASML EUV lithography under export controls
An inability to design competitive chips in-house
A lack of domestic demand for advanced chips
6. A hyperscaler weighs a custom ASIC against renting NVIDIA GPUs. The ASIC case is strongest when:
The workload is novel and changing rapidly each quarter
The workload is large, stable, and run at enormous volume
The team lacks any in-house compiler or software expertise
CUDA library coverage is the deciding purchase criterion
Part C — Synthesis Traces
These are the two recurring capstone exercises. Try to narrate each all the way through from memory — naming the bottleneck and the investable name at every hop — before revealing.
Trace 1 — One token, from sand to served output
Narrate the full path. At each hop: what happens, the bottleneck, and the name that captures the value.
The chain
1. Design. Chip is designed in EDA tools & IP. Bottleneck: tool/flow lock-in. Cadence, Synopsys, Arm.
2. Lithography. Patterns printed with EUV light. Bottleneck: EUV is a monopoly. ASML.
3. Fab. Wafer fabricated at the leading edge; yield is the learning curve. Bottleneck: process yield. TSMC.
4. HBM. Memory dies stacked with TSVs for TB/s bandwidth. Bottleneck: stacking yield. SK Hynix, Micron, Samsung.
5. Advanced packaging. GPU die(s) + HBM co-packaged on a CoWoS interposer. Bottleneck: CoWoS-L capacity. TSMC.
6. Accelerator + system. Becomes a B200, wired into an NVL72 rack via NVSwitch. Bottleneck: NVLink topology + CUDA. NVIDIA (AMD #2; Broadcom ASICs).
7. Kernel + serving. The token is generated: prefill (compute) then decode (bandwidth), KV cache held in HBM. Bottleneck: memory bandwidth at decode. NVIDIA software (TensorRT-LLM); vLLM.
9. Served to user. Token leaves the hyperscaler cloud. MSFT, Google, Amazon, Meta.
Trace 2 — One dollar of hyperscaler AI capex
Where does it land, and roughly what fraction does each layer capture? (Order-of-magnitude, not precise accounting.)
Where the dollar flows (rough shares of a GPU-server dollar)
The accelerator itself dominates — the GPU/system vendor captures the largest slice, and within the GPU's bill of materials the two biggest costs are HBM and TSMC wafer + CoWoS packaging. So a large share flows to NVIDIA (margin) and through it to TSMC and SK Hynix/Micron.
Power, cooling, facility — a growing slice as rack density hits 120 kW. Vertiv, Eaton, utilities, data-center REITs.
Upstream, invisibly: every chip in that dollar was designed on Cadence/Synopsys tools and printed on ASML EUV — they tax the whole flow regardless of who wins the GPU socket.
Key investor insight: the picks-and-shovels names (ASML, TSMC, EDA) get paid whether NVIDIA or AMD or a custom ASIC wins the accelerator socket. That's why they sit highest on the durability spectrum.
Part D — Score Yourself & What to Review
Count the Part A cards and Part C traces you could produce before revealing. Be honest — recognition is not recall.
If you stumbled on…
Re-read
The one idea to re-anchor
Cards 2, 7, 10 (inference, KV cache, dual-die)
L8, L15
Decode is bandwidth-bound; KV cache can exceed weights at long context
Cards 3, 4, 5 (scaling, memory, parallelism)
L11, L13
16 bytes/param and the TP-needs-NVLink physics
Cards 1, 6, 11 (TSMC, HBM, China)
L4, L12, L2/L3
Yield is an accumulated learning curve money can't shortcut
Cards 8, 9 (ASIC vs GPU, alignment)
L10, L14
Flexibility + CUDA are what the GPU keeps; alignment is <5% of compute
Card 12 or either trace
L1 + this checkpoint
The value chain in order, with the moat structure at each node
Next durable-judgment step. This checkpoint tested recall of mechanisms. The companion artifact, THESIS.md, turns those mechanisms into positions — per-name bull case, bear case, key technical assumption, and the falsifier that would change your mind. Open it, and for each name try to state the falsifier from memory before reading it.
From your instructor: Active recall feels worse than re-reading and works far better — if this was uncomfortable, it was working. Tell me which cards you missed and I'll build a short, targeted follow-up that re-tests only those, spaced a few days out. When you're ready for new material, the queue is L16 (AMD vs NVIDIA), L17 (ASML/EUV), or L18 (EDA) — all three close named-thesis gaps.