Lesson 11 · Chips & LLMs

Scaling Laws

Why each frontier model generation demands exponentially more chips — and how to calculate exactly how many

📍 Scaling laws · Kaplan → Chinchilla · 6ND formula · H100 translation · Test-time compute ~25 min

The Central Question

Why doesn't there come a day when the chips are "enough"?

Every year, hyperscalers announce ever-larger GPU clusters. Google guided $75B in capex for 2025. Microsoft committed $80B. Meta operates 350,000+ H100-equivalent GPUs. Investors who ask "when does this end?" are asking the wrong question. The correct question is: what is the mathematical relationship between compute and model quality?

The answer comes from a two-page empirical result published in 2020 — and a correction that arrived two years later. Together they explain why chip demand is not a cyclical trend but a mathematically predictable, compounding demand driver.

Core Thesis

Model quality (cross-entropy test loss) decreases as a smooth power law of training compute. This relationship has been verified across more than seven orders of magnitude of compute. There is no saturation point on the horizon. More compute → better model, always — the returns are diminishing but never zero. Scaling laws are the demand equation behind every GPU purchase order.

Section 1

What a power law means: diminishing but endless returns

The relationship between test loss L and training compute C (measured in FLOPs) follows:

L(C) ≈ (C₀ / C)^α α ≈ 0.05–0.07 across architectures. Each 10× increase in compute reduces loss by roughly 11–15%.

In log-log space this is a straight line. The practical implication: you can always buy a better model — it just gets more expensive. Going from GPT-2 quality to GPT-3 quality required ~1,000× more compute. Getting from GPT-3 to frontier-2024 quality required another ~100–1,000×. The law has not broken down at any scale tested.

Loss vs. Compute — Power Law (log-log, schematic)

A straight line in log-log space is a power law. The relationship holds from 10²⁰ to beyond 10²⁵ FLOPs with no sign of saturation. Chinchilla and GPT-3 have similar compute; the difference between them is how compute was allocated between model size and data — not total compute.

Section 2

The Kaplan → Chinchilla paradigm shift

Kaplan et al. (OpenAI, 2020) discovered the power law and drew an operational conclusion: for a fixed compute budget, prioritize model size. Scale parameters faster than data. GPT-3 followed this: 175B parameters on 300B tokens.

Hoffmann et al. (DeepMind, 2022) — the Chinchilla paper — ran a more controlled experiment with 400 models across a wider compute range and found Kaplan's conclusion was wrong. The correct allocation: model size and data tokens should scale equally. The empirical rule of thumb:

D_opt ≈ 20 × N Optimal training tokens ≈ 20 per parameter. GPT-3 (175B params) needed ~3.5 trillion tokens, not 300 billion — it was undertrained by 11×.

DeepMind trained Chinchilla — 70B parameters, 1.4T tokens — for the same compute as Gopher (280B params, 300B tokens). Chinchilla outperformed Gopher on every benchmark. Half the parameters, 20× more data, identical compute. This reshaped how every major lab trains frontier models.

2020

GPT-3

175B params
300B tokens
1.7 tok/param

undertrained data

2022

Chinchilla

70B params
1.4T tokens
20 tok/param

compute-optimal

2024

Llama 3 8B

8B params
15T tokens
1,875 tok/param

inference-optimal

Section 3

The 6ND formula: FLOPs for any training run

For a transformer with N parameters trained on D tokens, total training compute is:

C ≈ 6 × N × D Derived as: 2 FLOPs/MAC × 3 passes (1 forward + 2 backward) × N MACs/token. Accurate within 2× for standard transformers.

This formula is the Rosetta Stone between AI research and chip demand. Every time a lab publishes a model's parameter count and token count, you can compute the training budget — and from that, the H100 cluster required.

Model	N (params)	D (tokens)	C = 6ND (FLOPs)	Tokens / param	vs. Chinchilla
GPT-2	1.5B	40B	3.6×10²⁰	27	≈ optimal
GPT-3	175B	300B	3.2×10²³	1.7	11× too few tokens
Chinchilla	70B	1.4T	5.9×10²³	20	optimal ✓
Llama 2 70B	70B	2T	8.4×10²³	29	1.4× beyond optimal
Llama 3 8B	8B	15T	7.2×10²³	1,875	94× beyond optimal
Llama 3 70B	70B	15T	6.3×10²⁴	215	11× beyond optimal
Llama 3 405B	405B	15.6T	3.8×10²⁵	38	2× beyond optimal

Chinchilla Optimal Frontier — Model Size vs. Training Compute (log-log)

Chinchilla Frontier: Model Size vs. Training Compute (corrected)

Models above the frontier have too many parameters for their training data — overtrained on params, undertrained on data. Models below the frontier train smaller models on proportionally more data — the modern inference-optimized strategy. Chinchilla sits exactly on the frontier. Note GPT-3 is 11× above the optimal token count for its compute budget.

Section 4

From FLOPs to H100 clusters: the hardware translation

An NVIDIA H100 SXM delivers 989 TFLOPS BF16 peak. Real training runs achieve Model FLOPs Utilization (MFU) of 35–45% due to inter-GPU communication, gradient synchronization, and memory bottlenecks. Effective throughput: ~376 TFLOPS per H100. Use C ÷ (376 TFLOPS × training seconds) to find the required cluster size.

989 TFLOPS

H100 peak BF16

38%

Typical MFU

real training runs

376 TFLOPS

Effective per H100

at 38% MFU

3.25×10¹⁹

FLOPs/H100/day

376T × 86,400s

Section 5

The inference-optimal correction

Chinchilla defines "compute-optimal" as the (N, D) allocation that minimizes loss for a given training compute budget. This ignores the cost of serving the model after training.

If you deploy a model to millions of users, the total lifetime inference compute — billions of forward passes — typically exceeds training compute by 10:1 or more. A 2× larger model costs ~2× more per token to serve. This changes the optimization problem fundamentally.

The Llama 3 Strategy

Meta's Llama 3 8B was trained on 15T tokens — roughly 94× more data than Chinchilla-optimal for an 8B model. The result: an 8B model with benchmark quality comparable to much larger models trained with fewer tokens. Each inference call is 8–10× cheaper to serve than a Chinchilla-optimal 70B alternative. This is not a mistake — it's optimizing for the full cost equation (training + serving), not just training.

Investment implication: inference-optimized training is additive to chip demand. You train for longer (more training GPU-hours) AND you serve a model that generates more queries because it's cheaper per token. Both training and inference chip demand grow.

Section 6

Test-time compute scaling: the new demand axis

Scaling laws originally described training compute. A new scaling axis has emerged: inference-time compute ("test-time compute"). Instead of spending all compute on training a larger model, spend compute during inference to reason through the answer.

OpenAI's o1, DeepSeek-R1, and Claude extended thinking all exploit this. The model generates thinking tokens — an internal reasoning chain — before its final answer. More thinking tokens → better performance on hard tasks, following its own smooth power law.

MATH-500: Accuracy vs. Inference-Time Token Budget (schematic)

Direct answer

~50%

256 tokens

~68%

1K tokens

~80%

4K tokens

~90%

16K tokens

~96%

Schematic based on OpenAI's o1 scaling results. Performance scales smoothly with inference-time token budget — a power law orthogonal to training compute. Each 4× increase in thinking tokens yields roughly another quality step.

Why this matters for chip demand: even a fixed deployed model uses more GPU-hours when queries trigger long reasoning chains. A model generating 16,000 thinking tokens uses roughly 16× the inference compute of a direct-answer model. As AI agents proliferate — running multi-step reasoning loops — inference compute demand can grow faster than training compute demand, creating a demand lever entirely independent of new training runs.

Section 7 · Investment Lens

Four demand dynamics for the chip investment thesis

1 · Exponential Training Demand

Each frontier generation requires 5–10× more FLOPs than the previous. GPT-3 to GPT-4 (estimated) was ~70×. This translates directly to GPU cluster procurement. As long as scaling laws hold — and they hold to at least 10²⁵ FLOPs with no sign of breaking — each new model needs a meaningfully larger cluster.

2 · The Data Wall

Internet text ≈ 10–100T tokens of unique high-quality text. At Chinchilla-optimal scaling, we approach this limit in the next 1–3 model generations. Synthetic data (RLHF, self-play), video/multimodal, and domain-specific corpora become critical extensions. Data quality and curation become a moat.

3 · Inference Dominates Long-Term

By 2027, inference is projected at 60–70% of total GPU compute demand. Training is a one-time event; inference is the perpetual revenue stream. Test-time scaling amplifies this — each query uses more cycles. The inference chip demand curve is structurally steeper than training.

4 · TSMC as the Demand Anchor

Scaling laws provide mathematical grounding for "more chips forever." NVIDIA GPU wafers + Broadcom XPU wafers + AWS Trainium wafers — all manufactured on TSMC advanced nodes. Published model parameters and token counts let you calculate training FLOPs and derive the required cluster, giving a bottom-up demand estimate.

Key signals to watch

Hyperscaler capex guidance: when Google/Microsoft/Meta raise capex, they are signaling planned training runs. Each compute increase implies a proportional cluster size increase.
Model FLOPs disclosures: papers that disclose parameter count and token count let you compute C = 6ND. A 10× FLOPs increase is a 10× GPU-hours demand multiplier.
Chinchilla ratio of new models: models trained far below Chinchilla-optimal (Llama 3 pattern) signal inference-optimized strategy — more inference chip demand plus a likely larger training run planned for the next generation.
Test-time compute depth: average thinking tokens per query × daily active users × queries/user = inference chip demand. Watch for extended thinking becoming default in consumer AI products.

The Integrated Thesis

Scaling laws → training demand grows exponentially with each generation. Inference-optimal training → more total training compute, smaller deployed models, more inference queries. Test-time scaling → each query uses more inference compute. Data wall → synthetic training runs add new compute demand waves. All four dynamics point in the same direction: sustained, compounding chip demand with quantifiable mathematical underpinning.

Primary Sources

Knowledge Check — Scaling Laws

1. For a fixed training compute budget, what does Chinchilla prescribe as the optimal allocation between model size (N) and training tokens (D)?

Scale model size much faster than data: for each 10× compute increase, grow N by 8× and D by 2×, following Kaplan's empirical result from 2020

Scale both equally: both N and D should grow as the square root of compute, keeping D/N ≈ 20, so a fixed compute budget is split evenly across parameters and training tokens

Scale data much faster than model: for each 10× compute increase, grow D by 9× and N by 1×, maximizing token exposure rather than model capacity

There is no universal optimum: the correct N/D ratio varies by task domain and target deployment hardware, so compute allocation is empirically tuned per project

2. GPT-3 (175B params, 300B tokens) and Chinchilla (70B params, 1.4T tokens) have similar training compute budgets. What does this comparison reveal about GPT-3?

GPT-3 was compute-optimal per Chinchilla: the 1.7 tokens/param ratio was the correct allocation given the available compute in 2020, and Chinchilla simply benefited from more advanced hardware

GPT-3 was data-saturated: it had seen too many repetitions of its 300B-token dataset, degrading performance through overfitting that a smaller dataset would have avoided

GPT-3 was undertrained on data: at 175B params, Chinchilla scaling laws say it needed ~3.5T tokens for compute-optimal training, not 300B — it had 11× too few tokens for its parameter count

GPT-3 was overparameterized but nearly optimal: Chinchilla shows the loss difference is minor and mostly attributable to architectural improvements rather than the N/D ratio shift

3. Llama 3 8B was trained on 15T tokens — about 94× more data than the Chinchilla-optimal amount for 8B parameters. Why is this economically rational rather than a violation of scaling laws?

Meta believed Chinchilla's scaling exponents were specific to DeepMind's dataset and did not generalize; their own experiments showed data always dominates model size at any scale

Chinchilla optimizes training compute only; inference-optimal training minimizes total cost including billions of future inference calls — a smaller, overtrained model is far cheaper to serve at scale, making the higher training cost worthwhile

Meta had surplus GPU capacity from other projects and chose to use idle H100 time on additional training tokens rather than incurring the opportunity cost of wasted hardware

The 8B parameter architecture requires more tokens to converge than larger models because its smaller attention heads cannot capture long-range dependencies in fewer passes through the data

4. A hyperscaler announces a new model with an estimated 10× more training FLOPs than their previous generation. What is the correct way to translate this into GPU demand?

10× FLOPs always means 10× more GPUs in the cluster, since training time is fixed by business constraints and the only degree of freedom is cluster size

10× FLOPs equals exactly 10× more GPU-hours total, regardless of how it splits between cluster size and training duration — procurement scales proportionally, and in practice hyperscalers grow clusters to keep training time bounded

10× FLOPs requires more than 10× GPUs because large clusters suffer superlinear communication overhead, so effective MFU falls as cluster size grows beyond a few thousand chips

FLOPs and GPU procurement cannot be directly compared because hardware efficiency improvements — from H100 to B200 to future generations — absorb most of the compute increase without requiring proportionally more chips

5. What is the primary investment significance of test-time compute scaling (o1, DeepSeek-R1, extended thinking) for semiconductor demand?

Test-time scaling reduces overall chip demand because smarter models answer in fewer tokens on average, lowering total inference compute relative to generating many shorter responses with weaker models

Test-time scaling is primarily a software efficiency gain and does not create incremental hardware demand because the same GPU cycles that were used for training are simply reallocated to inference-time reasoning

Test-time scaling creates an independent demand axis: each deployed query now uses many more GPU-hours for its reasoning chain, meaning inference chip demand grows even without new model training runs — agentic workloads could make this the dominant demand driver

Test-time scaling favors TPUs over GPUs because systolic arrays are more efficient at the autoregressive token generation that long reasoning chains require, shifting inference demand away from NVIDIA's installed base

Questions worth exploring: "Why do scaling laws eventually break down, and do they?" · "What is the data wall and how close are we to hitting it?" · "How does reinforcement learning (RLHF, GRPO) fit into the scaling picture — does it follow the same laws?" · "Can test-time compute scaling replace training scaling indefinitely?" · "What happened with GPT-4's scaling — why didn't OpenAI publish a technical report?" Ask your teacher any of these to go deeper.

Coming Up — Lesson 12

HBM and SK Hynix: every accelerator in this course — H100, TPU, XPU — is memory-bandwidth-limited during inference. High Bandwidth Memory (HBM) is the critical enabler. We'll cover HBM architecture, how it achieves 3.35 TB/s bandwidth vs. DDR5's 89 GB/s, the SK Hynix / Micron duopoly, CoWoS integration, and why HBM yield limits how many H100s TSMC can ship per quarter.