RLHF — How Reinforcement Learning Aligns Language Models

Posts 1–6 built RL from the ground up; this post is its headline modern use. The reason RLHF matters here is that it’s where this series’ abstractions become the thing behind every chat assistant: the policy-gradient and actor-critic machinery (post 6), the reward design problem (post 2), and the exploration limits of real-world RL (post 5) all converge on one elegant solution.

The problem RLHF solves

A language model pretrained on the internet learns to predict text, which makes it fluent but not necessarily helpful. It will happily continue a prompt in a toxic direction, give a confidently wrong answer, or produce something unhelpful, because “predict the most likely next token” is not the same as “be a good assistant.” We want to steer it toward helpfulness, honesty, and harmlessness — but here’s the wall: you cannot write a reward function for those qualities. There’s no formula for “this response is helpful.” It’s a matter of human judgment, varying by context, impossible to specify in code.

This is exactly post 2’s reward-design problem at its hardest: the goal is real but unspecifiable. Supervised fine-tuning helps partially — show the model examples of good responses — but you can’t write enough examples to cover everything, and crucially, it’s often easier for a human to compare two responses (“this one’s better”) than to write the ideal response from scratch. RLHF is built on that insight: don’t hand-code the reward and don’t demand perfect example answers — learn the reward from human preference comparisons, then use RL to optimize the model against it.

How RLHF works: three stages

RLHF has a clear three-stage pipeline, and each stage maps onto something from this series:

1. Supervised fine-tuning (SFT). Start from a pretrained model and fine-tune it on a modest set of high-quality example responses, giving it a reasonable baseline of following instructions. This is ordinary supervised learning — the starting policy for the RL to come.

2. Train a reward model from human preferences. Collect human judgments in the easy form: show people pairs (or sets) of model responses to the same prompt and have them rank which is better. Then train a separate reward model — a neural network that takes a response and outputs a scalar “how good” score — to predict those human preferences. This is the crucial move: it turns fuzzy, unspecifiable human judgment into a computable reward function (post 2’s unsolvable problem, solved by learning the reward instead of writing it). The reward model is a learned stand-in for human judgment, able to score the millions of responses that humans never could.

3. Optimize the policy with RL. Now it’s a standard RL problem: the language model is the policy (the actor, post 6), generating responses is taking actions, and the reward model provides the reward. Use an actor-critic policy-gradient algorithm — in practice PPO (post 6) — to adjust the model’s parameters so it generates responses the reward model scores highly. The model explores (generates varied responses), gets scored, and policy-gradient updates make high-scoring behavior more likely. This is RL exactly as the series built it, with a learned reward and a language model as the agent.

A critical detail in stage 3: the optimization includes a constraint keeping the model close to its original behavior (a KL penalty against the pre-RLHF model). Without it, the model would reward-hack (post 2) — exploiting quirks of the reward model to score high while producing degenerate or gibberish text that games the imperfect reward. This is post 2’s reward-hacking warning made concrete at scale, and the constraint is the guardrail against it.

Why RLHF is the series’ capstone — and its limits

RLHF is where this entire series pays off, because every core idea reappears: - The reward-design problem (post 2) — “you can’t write a reward for helpfulness” — is the whole motivation, solved by learning the reward. - Reward hacking (post 2) is the central danger, countered by the KL constraint. - Policy gradients and actor-critic / PPO (post 6) are the optimization machinery. - The sample-inefficiency and safety limits of real-world RL (post 5) are why you can’t just let the model explore freely against the real world — you learn from a bounded set of human preferences instead, and keep the policy anchored.

It’s a beautiful demonstration that RL is not just for games: it’s the tool for optimizing any system toward a goal you can judge but not specify — and “be a good assistant” is exactly such a goal.

But RLHF has real limits worth knowing: the reward model is only as good as the human preferences it learned from, so it inherits the biases, inconsistencies, and blind spots of the labelers; it can over-optimize toward what sounds good to raters (sycophancy, confident-but-wrong answers) rather than what is good; and gathering quality human preference data is expensive. These limits drive active research into alternatives — learning from AI feedback (RLAIF), direct preference optimization (DPO) that skips the explicit reward model, and constitutional approaches that encode principles — but the core idea RLHF established stands: align a model by learning a reward from human judgment and optimizing with RL. That idea, built entirely from this series’ concepts, is what made the current generation of AI assistants possible.

The throughline: RLHF solves the problem that you can’t write a reward function for “helpful, honest, harmless” — post 2’s reward-design wall at its hardest — by learning the reward from human preference comparisons (easier to rank two responses than to write the ideal one). Its three stages: supervised fine-tuning (a baseline policy), train a reward model to predict human preferences (turning unspecifiable judgment into a computable reward), and optimize the policy with RL (the LLM is the actor, the reward model gives reward, PPO does the updates) — with a KL constraint keeping the model close to its original behavior to prevent reward hacking. It’s the series’ capstone (reward design, reward hacking, actor-critic/PPO, real-world RL limits all reappear) and proof that RL optimizes any goal you can judge but not specify — though it inherits labeler biases, can induce sycophancy, and is costly, driving alternatives like DPO and RLAIF.

Key takeaways

• RLHF exists because you can’t write a reward function for “helpful, honest, harmless” (post 2’s reward-design problem at its hardest) — so it learns the reward from human preference comparisons (ranking two responses is far easier than writing the ideal one).
• Three stages: (1) supervised fine-tuning (baseline instruction-following policy), (2) train a reward model to predict human preferences — turning fuzzy human judgment into a computable reward that can score millions of responses, (3) optimize the policy with RL where the LLM is the actor, the reward model gives the reward, and PPO (post 6) does the updates.
• A KL constraint keeps the tuned model close to its original behavior, preventing reward hacking (exploiting the imperfect reward model to score high with degenerate text) — post 2’s warning, concrete at scale.
• RLHF is the series’ capstone: reward design (post 2), reward hacking (post 2), actor-critic/PPO (post 6), and real-world RL’s sample/safety limits (post 5 — why you learn from bounded human preferences, not open exploration) all converge in it.
• Its limits: the reward model inherits labeler biases/inconsistencies, can induce sycophancy (optimizing what sounds good to raters), and needs expensive preference data — driving alternatives (DPO, RLAIF, constitutional methods) — but its core idea (learn a reward from human judgment, optimize with RL) made modern AI assistants possible.

Further reading

• Reinforcement learning from human feedback — aligning models to human preference
• Proximal policy optimization — the RL algorithm RLHF uses