HBM & SK Hynix

Why memory, not compute, is the constraint on AI performance

In Lesson 8 we established that LLM inference has two phases: prefill (compute-bound) and decode (memory-bandwidth-bound). The decode phase — generating each new token — requires loading the entire model weight matrix from memory to compute units, one token at a time. A 70B parameter model in FP16 occupies 140 GB of memory. For each generated token, 140 GB must stream from memory to the GPU's arithmetic units.

Standard server DRAM (DDR5) delivers ~89 GB/s per channel. At 8 channels, that is ~700 GB/s — not enough to stream 140 GB in a tolerable latency. The H100 solves this with HBM3: 3.35 TB/s bandwidth, nearly 5× the best DDR5 configuration. Without HBM, large language model inference would be economically unviable.

The memory wall: compute scales faster than bandwidth

GPU compute throughput (TFLOPS) has grown roughly 3× per year since 2016. Memory bandwidth has grown roughly 1.5× per year. This widening gap — the "memory wall" — is the fundamental reason why memory bandwidth, not raw compute, determines inference performance.

The ridge point is where compute and memory bandwidth are equally the limiting factor. At H100's ridge point of 295 FLOPs/byte, operations that use more FLOPs per byte of memory accessed are compute-bound; operations that use fewer are memory-bandwidth-bound. LLM decode at 1–2 FLOPs/byte is ~150–295× below the ridge point — it sits almost entirely in memory-bound territory.

Why standard DRAM can't solve this

DDR5 (server DRAM) achieves high throughput by running many DRAM chips in parallel. A typical 8-channel server memory system peaks at ~700 GB/s. The limit is the bus width: each DIMM slot connects via a 64-bit data bus, and multiplying bus width requires physically wider PCB traces and larger connectors — the chip and board get prohibitively large.

GDDR6X (used in consumer RTX GPUs) pushes the bandwidth higher via a wider on-die bus, but faces power limits: high-speed long copper traces dissipate significant energy. An RTX 4090 has 21.8 Gbps GDDR6X bandwidth (576 GB/s) — less than HBM2e, and at nearly double the power draw per GB/s of HBM.

HBM architecture: vertical stacking + silicon interposer

HBM solves the bus-width problem with two innovations: through-silicon vias (TSVs) and the silicon interposer.

How TSVs enable the 1,024-bit bus

Through-silicon vias are vertical copper pillars, 1–10 µm in diameter, that are etched and filled through the silicon die. A single HBM3 stack contains ~1,024 data TSVs per die — 1,024 independent electrical connections running vertically through each layer. This is physically impossible with horizontal wire bonding, which is limited to the die's periphery. TSVs access the die's entire area for I/O.

The silicon interposer brings the GPU die and HBM stacks close enough (< 5 mm of trace length) that the 1,024-bit bus can run at multi-gigabit-per-pin speeds without the signal integrity problems that afflict long copper PCB traces. Shorter distance = less capacitance = lower power = higher speed.

HBM generation evolution: bandwidth doubles each cycle

CoWoS integration: why HBM yield constrains GPU supply

TSMC's CoWoS (Chip on Wafer on Substrate) process assembles the GPU die and HBM stacks on a silicon interposer. This is a two-stage manufacturing process:

1. Die preparation: NVIDIA ships GPU dies (fabricated on TSMC N4/N5); SK Hynix/Micron ship pre-tested HBM stacks.
2. CoWoS assembly: TSMC places GPU die + HBM stacks on the interposer, runs solder reflow, and tests the assembled module. If any HBM stack fails integration testing, the entire module is scrapped — the GPU die included.

Market structure: SK Hynix's structural advantage

Why SK Hynix's lead is durable

HBM manufacturing requires skills that take years to develop:

• TSV etching precision: TSV holes must be etched through silicon to within nanometer tolerances. Misalignment destroys the die. SK Hynix has been manufacturing TSVs since 2013 (the original HBM1).
• Thermal management: stacking 12 DRAM dies creates severe heat density. SK Hynix developed proprietary underfill materials and stack bonding techniques that prevent delamination under thermal cycling.
• Micro-bump yield: each die-to-die connection uses thousands of micro-bumps (~20 µm pitch). A single failed micro-bump degrades the entire stack. This is SK Hynix's most guarded process know-how.
• NVIDIA co-development: SK Hynix engineers are embedded in NVIDIA's GPU architecture team. HBM3e specifications were developed jointly. Samsung was not part of this process.

ASP premium: HBM3e sells for roughly 3–5× the price per gigabyte of standard DDR5. A 192 GB HBM3e configuration (B200) represents approximately $15,000–20,000 of HBM value — about 40–60% of the total module cost. At 2+ million B200 modules (NVIDIA's estimated 2025 shipments), SK Hynix HBM3e revenue from NVIDIA alone could exceed $20B.

Investment implications

Key signals to watch

• NVIDIA certification announcements: each new HBM supplier qualification reduces SK Hynix's pricing power. Track official NVIDIA statements on HBM3e/HBM4 supplier approval.
• SK Hynix HBM revenue mix: watch quarterly earnings for HBM as % of total DRAM revenue. Rising HBM mix → rising gross margin → supports valuation.
• HBM content per GPU: B200 at 192 GB (vs. H100 at 80 GB) means 2.4× more HBM revenue per GPU sold, even at flat ASP. GPU volume × HBM GB per GPU = addressable HBM market size.
• CoWoS capacity at TSMC: TSMC CoWoS advanced packaging is a separate supply constraint. In 2024, CoWoS capacity limited H100 module assembly even when GPU dies and HBM stacks were available.

Recommended reading

• SK Hynix Earnings Calls — Management explicitly discusses HBM mix, yield, and ASP trends every quarter. The FY2024 calls are the clearest primary source on HBM market dynamics from the supplier perspective.
• JEDEC HBM Standards — JEDEC is the standards body that defines HBM specs. Understanding bus width, transfer rate, and die stacking specs at the source, not from secondary summaries.
• Efficiently Scaling Transformer Inference (Pope et al., Google, 2022) — The roofline model analysis applied to transformer inference. The paper that formally established why inference is memory-bandwidth-bound and derived the arithmetic intensity of decode. Directly connects to the investment thesis for HBM.