Lesson 21 · Chips & LLMs · Capstone arc (1 of 3)

Inference Unit Economics

Everything you've learned — compute (L8), HBM (L12), and power (L20) — collapses into one number you can defend to a skeptic: the cost to serve a million tokens. Build it from the metal up, then watch it decide build-vs-buy and who captures the margin.

Builds on: L8 (inference), L12 (HBM), L20 (power) Skill: model $/1M tokens and read MaaS margin

This is the lesson where the course becomes operational. You've assembled the whole stack; now you compress it into the master metric of the inference business: cost per million tokens ($/Mtok). Every API price, every "our model is cheaper" claim, every build-vs-buy decision, and the entire margin structure of AI services reduces to this number versus the price charged for it. The skill: build it bottom-up so you can spot which inputs actually move it — and therefore which companies capture the value.

Core thesis: Serving cost is cost-per-GPU-hour ÷ tokens-per-hour. The numerator is set by hardware capex + power (L20); the denominator is set by throughput, which is gated by memory bandwidth and HBM capacity (L8, L12). The gap between this cost and the API sell price is the Model-as-a-Service (MaaS) gross margin — enormous for frontier models, near-zero for commodity ones. The same math sets the build-vs-buy line (own ASICs/GPUs vs rent) — and that line is exactly where Broadcom's ASIC case, AMD's TCO wedge, and NVIDIA's pricing power are won or lost.

01 — The Master Formula

Two Numbers, One Metric

Strip away the noise. The cost to generate tokens is just a rate of spending divided by a rate of producing:

$/Mtok = ( cost per GPU-hour ÷ tokens per second ÷ 3600 ) × 1,000,000 Numerator = what an hour of the machine costs you. Denominator = how many tokens that hour produces. Everything else is detail feeding these two.

The art is in those two inputs, and each maps directly to a prior lesson:

Cost per GPU-hour — if you rent, it's the cloud price (~$3/hr H100, ~$6/hr B200 in 2026 [index]). If you own, you build it from amortized capex + power (L20: GPU watts × PUE × electricity price).
Tokens per second — throughput, set by the roofline (L8): for token-by-token decode, inference is memory-bandwidth-bound, so HBM bandwidth and capacity (L12) — not raw FLOPS — usually cap it. More HBM also lets a big model fit in fewer GPUs, raising effective throughput per dollar.

02 — Build It Yourself

The Live Model: Rent vs. Own

Move the sliders. The left column is the rent path (cloud price, all-in). The right is the own/build path, assembled from capex + power exactly as in L20. Watch what happens to $/Mtok, to gross margin against your sell price, and to the build-vs-buy verdict.

Inference $/Mtok & Build-vs-Buy Calculator

Defaults model a ~70B-class model served with continuous batching + FP8 on one high-end GPU. All figures editable.

Throughput 1600 tok/s

Sell price (output) $5.00/Mtok

RENT — cloud price $3.00/GPU-hr

OWN — server capex $30k/GPU

OWN — useful life 3.0 yrs

OWN — utilization 70%

OWN — power draw 1.2 kW/GPU

OWN — PUE × elec $/kWh 1.3 × $0.08

RENT — cost to serve

$0.52

per 1M tokens · margin 90%

OWN — cost to serve

$0.31

per 1M tokens · margin 94%

Note how small the power line is per token at these prices (a few cents per Mtok) yet how it dominates the aggregate bill at gigawatt scale — that's the L20 paradox: negligible per token, decisive per cluster.

03 — Reading the Three Levers

What Actually Moves the Number

Lever	Pulls $/Mtok via…	Owned by (thesis name)
Throughput ↑	Bigger denominator: continuous batching, FP8/FP4 quant, MoE sparsity, speculative decoding	NVDA (software), the model lab
HBM bandwidth/capacity ↑	Lifts the memory-bound ceiling (L8) & fits the model in fewer GPUs	SK Hynix; AMD (capacity wedge)
$/GPU-hour ↓	Cheaper capex/power per hour: own vs rent, ASICs, lower PUE	AVGO (ASIC), utilities/cooling (L20)

The headline trend: GPT-4-equivalent serving fell from ~$20/Mtok in late 2022 to ~$0.40/Mtok by 2026, and API prices dropped ~80% in a single year. [Epoch AI] That deflation is all three levers compounding — and it's why the volume of tokens matters as much as the margin per token.

04 — Build vs. Buy: Where the Thesis Lives

When Owning Silicon Beats Renting

The calculator's verdict line is the heart of the investment case. Owning (GPUs or custom ASICs) trades a big fixed capex for a low marginal cost; renting is pure variable cost. The cross-over depends on utilization and workload stability:

High, stable volume → own (and increasingly, own an ASIC). A custom ASIC (Broadcom-designed TPU/MTIA) strips out GPU flexibility you don't need, cutting capex and power per token. This is why Google, Meta, and Amazon build silicon — and why AVGO gets paid as the merchant ASIC arms dealer (L10).
Spiky or experimental volume → rent. You can't fill an owned cluster, so the GPU cloud's flexibility wins despite its markup — and that markup is NVDA's pricing power flowing through the cloud's bill.
Memory-bound, large-model serving → the HBM-capacity wedge. Fitting a model in fewer chips lowers $/Mtok directly — exactly AMD's MI300X/MI350X argument (L16), and pure demand for SK Hynix HBM (L12).

Diligence reframe: When a company claims an inference cost advantage, decompose it into the three levers and ask which one and is it durable. A throughput edge from a software trick (batching) is copyable in months; a structural edge from owning cheaper silicon + power at high utilization is not. That distinction is the difference between a feature and a moat.

05 — The Margin Question

Who Captures the Spread?

The MaaS gross margin is (sell price − serve cost) ÷ sell price. At a $0.52/Mtok serve cost and a $15/Mtok frontier output price, that's a ~97% gross margin — extraordinary, and the reason model labs and clouds raced into the business. But two forces compress it:

Commoditization at the low end: open-weight models served at $0.30–$0.50/Mtok leave almost no spread — a race to serve cost.
The value migrates to whoever owns the scarcest lever. If serving is memory-bound, margin accrues to HBM (SK Hynix) and the accelerator (NVDA/AMD/AVGO); if it's power-bound at scale, to the power layer (L20). The model lab keeps margin only where the model itself is differentiated.

Tie to your thesis: this is the quantitative spine of the whole portfolio. NVDA's gross margin is a line item in every renter's serve cost — durable only while throughput-per-dollar leadership holds. AVGO's ASIC TAM is exactly the set of workloads where the build-vs-buy line tips to "own." AMD's wedge is the memory-bound segment where capacity, not topology, decides $/Mtok. And the inference price deflation is the demand engine: cheaper tokens → more tokens → more silicon and power. Add to THESIS.md a per-name note: which lever of $/Mtok does this name own, and how copyable is it?

Primary Source

Go Deeper

Read first: Introl — "Inference Unit Economics: The True Cost Per Million Tokens" for the full bottom-up build. Then: Epoch AI's inference price-trend data for the deflation curve, and SemiAnalysis (in RESOURCES.md) for serve-cost teardowns by GPU generation.

Comprehension Check

Quiz — 5 Questions

Select the best answer for each.

1. The cost to serve a million tokens is fundamentally:

The price of the GPU divided by the model's parameter count

Cost per GPU-hour divided by the tokens produced per hour

The electricity bill of the datacenter divided by user count

The API sell price minus the model provider's gross margin

2. For token-by-token decoding, throughput is usually capped by:

Raw FLOPS, because each token needs a huge matrix multiply

Network latency between the user and the datacenter server

Memory bandwidth and HBM capacity, making it memory-bound

The clock frequency of the host CPU orchestrating the run

3. A custom ASIC tends to beat renting GPUs specifically when the workload is:

Small, spiky, and changing model architectures every few weeks

Run on a laptop with no access to any datacenter hardware

High-volume and stable, so the fixed capex is fully utilized

Purely experimental research with no production traffic at all

4. An inference cost edge is most likely a durable moat when it comes from:

A continuous-batching software trick that competitors can copy

Owning cheaper silicon and power at high, sustained utilization

A one-quarter promotional discount on the cloud GPU rate

A marketing claim of leadership without published throughput

5. As inference prices fell from ~$20 to ~$0.40 per Mtok, the effect on hardware demand was:

Demand fell, since each token now earns far less revenue

Demand was unchanged, as price and volume are unrelated here

Demand rose, as cheaper tokens drove far higher token volume

Demand moved entirely to CPUs, which became cheaper to run

From your instructor: The skill to keep — collapse any inference claim into cost-per-GPU-hour ÷ tokens-per-hour, then ask which of the three levers (throughput, HBM, $/GPU-hr) is doing the work and whether it's copyable. That single decomposition tells you the serve cost, the MaaS margin, the build-vs-buy line, and which thesis name captures the spread. Ask me anything: how continuous batching lifts throughput, why decode is memory-bound, or how to compute the ASIC break-even utilization precisely.