Lesson 14 · Chips & LLMs

Post-Training Alignment

RLHF, DPO, and GRPO — how a raw next-token predictor becomes a useful assistant, and why alignment compute is a growing demand category

📍 SFT · RLHF · Reward model · PPO · DPO · GRPO · Verifiable rewards ~25 min

The Central Problem

Pretraining produces a parrot. Alignment produces an assistant.

In Lesson 7 and Lesson 11 we covered how transformer pretraining works: minimize the cross-entropy loss on predicting the next token in a vast text corpus. A pretrained model is extraordinarily good at this task — but "predict what comes next on the internet" is not the same as "be helpful, accurate, and safe."

A raw pretrained model will complete any prompt plausibly. Ask it "how do I make a bomb?" and it will answer, because that text pattern appears in its training corpus. Ask it to write a cover letter and it will generate a plausible example — but might hallucinate credentials, change tone mid-paragraph, or ignore your instructions entirely. Post-training alignment bridges this gap.

Core Thesis

Post-training alignment is a separate phase of training that costs 1–5% of pretraining compute but can dramatically change a model's usefulness and safety. It encompasses three main techniques: Supervised Fine-Tuning (SFT) on demonstrations, Reinforcement Learning from Human Feedback (RLHF) with a reward model, and newer methods like DPO and GRPO that achieve similar outcomes more efficiently. As models become more capable through pretraining, alignment compute — not pretraining compute — increasingly determines which models are deployable and competitive.

Section 1

Step 1: Supervised Fine-Tuning (SFT)

The first stage of alignment is conceptually simple: train the model on a curated dataset of high-quality instruction-following examples. Each example is a (prompt, ideal response) pair, written or verified by human annotators.

SFT uses the same next-token-prediction loss as pretraining, but the loss is computed only on the response tokens (not the prompt). This teaches the model the format and style of being a helpful assistant, without yet teaching it to prefer some responses over others.

SFT alone is insufficient. Given a prompt, there are many valid responses — some better than others. SFT teaches what kind of responses to give but not which of many plausible responses is best. That is the job of the reinforcement learning stage.

Section 2

RLHF: the three-stage InstructGPT pipeline

InstructGPT (Ouyang et al., OpenAI, 2022) introduced the RLHF pipeline that became the foundation for ChatGPT, Claude 1–2, and most major aligned models. It has three stages:

Stage 1

SFT

Fine-tune pretrained model on ~13K human-written (prompt, response) demonstrations.

Low compute: ~0.1% pretraining

Stage 2

Reward Model

Train a separate model to predict human preferences. Given two responses (A, B) to the same prompt, output a scalar reward score. ~33K comparison pairs.

Medium: ~0.5% pretraining

Stage 3

RL with PPO

Use the reward model to score the SFT model's outputs. Run PPO to optimize the model toward high-reward outputs while penalizing divergence from SFT baseline.

High: 1–5% pretraining. 4 models in memory.

Stage 2 in depth: how the reward model is trained

The reward model (RM) takes a prompt + response and outputs a scalar reward. It is trained on pairwise preference data: human labelers see the same prompt with two responses and pick which is better.

The RM loss is a Bradley-Terry preference model: maximize the probability that the preferred response gets a higher score than the rejected one. Given preferred response y_w and rejected response y_l:

L_RM = −E[log σ(r(x, y_w) − r(x, y_l))] σ = sigmoid. r(x, y) = reward score. Minimized when r(preferred) >> r(rejected).

Stage 3 in depth: PPO and the 4-model problem

Proximal Policy Optimization (PPO) treats the language model as a policy (maps prompt → response token distribution) and optimizes it to maximize reward while staying close to the SFT baseline. This requires four models simultaneously:

Policy model (being trained): the LM generating responses
Reference model (frozen SFT model): the KL-divergence anchor — prevents reward hacking
Reward model (frozen RM): scores generated responses
Value function (critic): estimates expected future reward to compute advantages

The KL penalty is critical: without it, the policy quickly discovers degenerate reward-hacking strategies — repetitive but highly-scored text, sycophantic flattery, or token sequences that fool the reward model. The full PPO objective is:

L_PPO = E[r(x, y) − β · KL(π_θ || π_ref)] r = reward model score. β = KL penalty weight. KL penalizes diverging too far from the SFT reference model.

RLHF's Core Problem: Reward Hacking

The reward model is a proxy for human preferences, not human preferences themselves. A sufficiently capable model can find inputs that score high on the reward model without actually satisfying users — this is "reward hacking" or "specification gaming." The KL penalty slows this down but doesn't eliminate it. After enough PPO steps, reward keeps rising but human-judged quality plateaus or falls. Practitioners call this the "reward model overoptimization" problem.

Section 3

DPO: eliminating the reward model entirely

Direct Preference Optimization (Rafailov et al., Stanford, 2023) proved that the RLHF objective — reward maximization subject to KL constraint — can be reparameterized as a direct supervised learning problem on the language model, with no separate reward model and no PPO.

The key insight: under the KL-constrained RLHF framework, the optimal policy implicitly defines a reward function. DPO directly optimizes this implicit reward by training on preference pairs:

L_DPO = −E[log σ(β · log(π_θ(y_w|x)/π_ref(y_w|x)) − β · log(π_θ(y_l|x)/π_ref(y_l|x)))] y_w = preferred response, y_l = rejected. β = temperature. Encourages higher relative probability for preferred vs. reference; lower for rejected vs. reference.

In plain terms: DPO trains the policy to assign relatively higher probability to preferred responses compared to the reference model, and relatively lower probability to rejected responses. No reward model training, no PPO, no critic — just two models (policy + frozen reference) and a standard gradient descent step.

RLHF vs. DPO: Architecture Comparison

DPO collapses the RLHF pipeline from 4 models + PPO to 2 models + supervised loss. It is mathematically equivalent to RLHF under certain assumptions, but far simpler in practice. Most open-source fine-tuning now uses DPO or its variants (IPO, KTO, ORPO).

Section 4

GRPO: reasoning alignment without a critic

Both RLHF and DPO train on human-labeled preferences — a labeler reads two responses and picks the better one. This is expensive, slow, and limited to domains where humans can evaluate quality. For tasks like mathematics, coding, and formal reasoning, there is a better signal: ground-truth verifiability.

Is the answer to this math problem correct? Does this code pass all unit tests? These questions have unambiguous answers — no human labeler needed. DeepSeek's GRPO (Group Relative Policy Optimization, 2024), used to train DeepSeek-R1, exploits this with a key simplification over PPO:

The GRPO insight: replace the critic with group sampling

PPO requires a value function (critic) to estimate baseline expected returns — so it can compute advantages (A = R − V(s)) and reduce variance in gradient estimates. Training this critic requires a fourth model in memory and adds significant complexity.

GRPO eliminates the critic. Instead, for each prompt, sample G responses from the current policy and use the group mean reward as the baseline:

A_i = (R_i − mean(R₁…R_G)) / std(R₁…R_G) Advantage of response i = how much better it is than the average response for this prompt. No critic model needed.

GRPO: Group Sampling for Advantage Estimation (G=6 responses)

GRPO samples G responses to the same prompt, scores them with a verifiable reward (math answer correctness, code test pass rate), and uses the group mean as the baseline. Responses above average get positive advantage (model is trained to generate them more); responses below average get negative advantage. No critic model needed — the group statistics replace it.

Verifiable rewards: the clean signal

GRPO's other breakthrough is using rule-based, verifiable rewards instead of a learned reward model. For mathematical reasoning:

Format reward: does the answer follow the expected format? (e.g., `\boxed{answer}`) → +0.5 if yes
Correctness reward: does the final answer match the ground-truth solution? → +1.0 if yes

These rewards require no human labelers and cannot be "hacked" by a learned reward model — the math answer is either correct or it isn't. DeepSeek-R1 was trained almost entirely on math and code with verifiable rewards, achieving o1-level performance at a fraction of the compute and without OpenAI's preference labeling infrastructure.

Section 5

Alignment methods compared

Method	Models in memory	Data required	Compute vs. pretraining	Reward hacking risk	Best for
SFT	1 (policy)	Human demonstrations	~0.1%	None (supervised)	Format learning, baseline capability
RLHF + PPO	4 (policy + ref + RM + critic)	Pairwise preferences + demos	1–5%	High — RM is imperfect proxy	General helpfulness, safety, diverse tasks
DPO	2 (policy + ref)	Pairwise preferences	0.5–2%	Low — no reward model to game	Open-source models, helpfulness, factuality
GRPO	3 (policy + ref + RM) or 2 (verifiable)	Prompts + verifiable answers (math/code)	1–3%	Very low (verifiable) — rule-based reward	Mathematical reasoning, coding, structured output
Constitutional AI / RLAIF	2–3 (policy + RM) with AI-generated data	AI-generated preference data (no humans)	1–3%	Medium — AI can have systematic biases	Safety, harmlessness, scalable feedback

Section 6 · Investment Lens

Investment implications

Alignment compute as a growing category

As pretraining data approaches natural limits (the data wall from Lesson 11), alignment and post-training compute become the primary differentiator between models. Labs run multiple alignment iterations on the same base model — RLHF round 1, RLHF round 2, safety fine-tuning, etc. Each round uses GPU compute. "More alignment" is becoming a competitive strategy distinct from "more pretraining."

DeepSeek's efficient alignment thesis

DeepSeek-R1 trained with GRPO + verifiable rewards achieved o1-level reasoning at a reported 5–10% of OpenAI's compute budget. This is primarily an alignment-efficiency improvement, not a pretraining efficiency improvement. If GRPO-style techniques generalize beyond math/code, they reduce the GPU-hours required per alignment iteration — potentially a headwind for inference GPU demand from alignment workloads.

Human data infrastructure

RLHF requires high-quality pairwise preference labels — expensive, slow, and domain-specific. Companies like Scale AI and Surge AI are picks-and-shovels plays on alignment data rather than alignment compute. RLAIF (AI-generated feedback) reduces but does not eliminate human label demand — humans are still needed to bootstrap the initial preference signal and audit AI-generated data quality.

Synthetic data loop

Modern alignment increasingly uses synthetic data: generate responses → verify with automated tools → train on verified responses. This creates a self-improving loop that can run continuously, generating GPU demand for both inference (generating synthetic data) and training (fine-tuning on it). The synthetic data flywheel means alignment compute scales with deployed model usage, not just with training runs.

Signals to watch

GRPO / verifiable reward adoption: if labs announce reasoning model training on non-math/code domains with verifiable rewards (e.g., scientific reasoning, logic), this signals a step-change increase in automated alignment compute without proportional human labeling cost.
Constitutional AI scaling: Anthropic's Constitutional AI uses AI-generated critiques to replace human preference labels at scale. Watch for public descriptions of Claude's training compute allocation between pretraining and alignment.
Scale AI / Appen financial results: as bellwethers for human data annotation demand. Revenue growth signals labs are still buying human preference data; stagnation signals RLAIF/synthetic data is displacing human labels.
Alignment compute disclosures: papers that explicitly report RLHF/DPO/GRPO compute as a fraction of pretraining — this lets you estimate the total compute per model release including alignment phases.

Primary Sources

Knowledge Check — Post-Training Alignment

1. Why does a pretrained language model need post-training alignment rather than being deployed directly after pretraining?

Pretrained models are computationally inefficient at inference time because their weights are optimized for gradient flow during training rather than for forward-pass throughput — alignment fine-tunes the weights to reduce inference latency

Pretraining produces a model that only generates tokens from its training distribution; alignment teaches the model to generalize to out-of-distribution prompts that users submit in production which weren't present in the pretraining corpus

Pretraining optimizes for next-token prediction on internet text, which includes harmful, wrong, and incoherent content — the model will complete any prompt plausibly, including requests for dangerous information; alignment teaches it to distinguish between what it can predict and what it should say

Pretrained model weights are stored in FP32 precision which is too memory-intensive for deployment; alignment fine-tuning converts the model to BF16 inference precision while preserving the knowledge encoded during pretraining

2. In RLHF, why is the KL divergence penalty between the policy and the reference model necessary?

The KL penalty is a regularization term that prevents the optimizer from overfitting to the small ~33K preference dataset used for reward model training, analogous to L2 weight decay in supervised learning

KL divergence measures the difference between the policy's probability distribution and the human preference distribution directly — the penalty ensures the model's outputs remain within the distribution that human labelers were recruited to evaluate

Without the KL penalty, PPO optimization finds reward-hacking strategies — outputs that score high on the imperfect reward model proxy but do not actually satisfy users — by drifting arbitrarily far from the SFT reference model; the penalty constrains how far the policy can deviate from the reference distribution per training step

The KL penalty compensates for the value function's estimation error in the critic model — without it, PPO's advantage estimates are biased toward high-variance trajectories that the critic assigned incorrect baseline values during early training

3. DPO is described as "mathematically equivalent to RLHF under certain assumptions." What does it actually eliminate that makes it simpler and more stable?

DPO eliminates the need for a reference model by incorporating the SFT behavior directly into the policy initialization, so the divergence penalty is implicit in the starting point rather than computed as a separate forward pass during training

DPO eliminates the separate reward model training stage and the PPO optimization loop entirely — it reparameterizes the RLHF objective so that preference data directly supervises the language model, removing the need for a scalar reward signal and the critic model that estimates expected returns

DPO eliminates pairwise preference data requirements — it can be trained on single-response quality annotations (good vs. bad) rather than pairwise comparisons (A is better than B), reducing the labeling cost and allowing larger datasets to be constructed from existing human feedback

DPO eliminates the need for gradient checkpointing by computing the alignment objective in a single forward pass without storing activations, reducing memory pressure compared to PPO's multi-step rollout that requires storing the entire generation trajectory

4. GRPO replaces the PPO critic (value function) with group-based advantage estimation. What is the practical significance of this for AI training infrastructure?

Eliminating the critic reduces training instability significantly — PPO critic updates were the primary source of training divergence in large models, and GRPO's stable group-mean baseline allows higher learning rates that converge 5× faster than PPO for the same final reward

Eliminating the critic removes one of the four models required in PPO — reducing from 4 to 3 models in memory — and enables GRPO to use verifiable rule-based rewards rather than a learned reward model, which together allow reasoning model training without expensive preference labeling infrastructure or the reward model training stage

Group sampling in GRPO allows TP-style parallelism across the G sampled responses — each response is generated on a different GPU simultaneously — creating a new distributed training topology that scales inference and training together in a single cluster configuration

GRPO's group baseline is computed from G samples of the same prompt rather than from a separate model, which allows it to train on a smaller dataset than PPO since each prompt effectively generates G labeled examples rather than requiring G separate prompts with independent human preference annotations

5. An investor reads that DeepSeek-R1 achieved o1-level reasoning at "5–10% of OpenAI's estimated compute budget." What is the most accurate interpretation of this for semiconductor demand?

Strongly bearish for GPU demand: if alignment efficiency improves by 10–20× across the industry, frontier model training will require 10–20× fewer GPUs per capability level — implying hyperscaler capex could fall dramatically as the same reasoning quality becomes achievable with much smaller training clusters

Neutral for GPU demand: the 10× efficiency gain is offset by a 10× increase in model deployments — more efficient alignment means more companies can afford to train aligned models, and the total GPU-hours consumed across the industry stays constant as supply meets the lower per-unit cost

Primarily applies to post-training alignment compute, not pretraining: DeepSeek's efficiency gain is in the RLHF/GRPO phase (~1–5% of total compute), not in the pretraining phase (~95%). The 10× alignment efficiency doesn't change the pretraining compute requirements that drive the majority of GPU demand — and cheaper reasoning capability expands deployment, increasing inference demand through Jevons paradox

Strongly bullish for GPU demand: DeepSeek proved that high-quality reasoning models can be trained at lower cost, which will democratize access to reasoning model training and cause dozens of new labs to train their own reasoning models — driving total GPU-hours consumed much higher than if only a few well-funded labs could afford it

Questions worth exploring: "What is Constitutional AI and how does it scale RLHF without human labelers?" · "How does RLAIF (RL from AI feedback) work and what are its limitations?" · "Why does reward hacking become more severe as the policy becomes more capable?" · "What is the relationship between GRPO and the AlphaGo self-play training approach?" · "How does post-training alignment interact with quantization — does aligning a BF16 model survive INT4 compression?" Ask your teacher any of these to go deeper.

Coming Up — Lesson 15

NVIDIA Blackwell: B200 architecture with first principles. Now that you understand the full training and inference stack — scaling laws, distributed parallelism, alignment, HBM, CoWoS — we can evaluate Blackwell with genuine depth. What does the NVL72 rack topology actually unlock for tensor parallelism? Why is the B200's memory bandwidth 8 TB/s instead of 3.35 TB/s, and what inference use case demands this? And what would it actually take for AMD MI350 or a Broadcom XPU to replace NVIDIA in a hyperscaler's training cluster?