Reinforcement learning from first principles — the third ML paradigm and the backbone of RLHF, built from the ground up. What RL is (learning to act by interacting with an environment to maximize cumulative reward — no answer key; the agent-environment-reward loop, the policy, delayed reward and credit assignment, exploration vs exploitation), Markov decision processes (the formal frame — states, actions, transitions, reward, discount; the Markov property; reward design and reward hacking as RL's decisive difficulty), value functions and the Bellman equation (estimating long-term value; V and Q functions; the recursive Bellman relation that solves credit assignment; planning vs learning), Q-learning and temporal-difference learning (learning value from experience, model-free, bootstrapping; off-policy learning, ε-greedy, and why the Q-table can't scale), deep reinforcement learning (neural function approximation; DQN from pixels, AlphaGo; instability and the fixes — experience replay, target networks; sample-inefficiency), policy gradients and actor-critic (learning the policy directly for continuous and stochastic control; high variance; actor-critic and PPO — the bridge to RLHF), RLHF (you can't write a reward for 'helpful, honest, harmless', so learn one from human preferences and optimize with PPO; the three-stage pipeline; the KL constraint against reward hacking; limits like sycophancy), and when to use RL (where it shines — games, control, alignment — and where a simpler method is the honest choice). Grounded in Wikipedia (reinforcement learning, multi-armed bandit, Markov decision process, Bellman equation, dynamic programming, Q-learning, temporal difference learning, deep reinforcement learning, proximal policy optimization, RLHF).
Supervised learning needs a teacher who knows every right answer; reinforcement learning needs only a goal and the freedom to try. It's how a system learns to play a game, control a robot, or — increasingly — align a language model, not by being shown correct outputs but by acting, seeing what happens, and learning from the consequences. This series builds RL from first principles, and it starts with why "learning from reward" is a fundamentally different and more general kind of learning.
Supervised learning needs a teacher who knows every answer; RL needs only a goal and the freedom to try. Learning from consequences via the agent-environment-reward loop, the policy, delayed reward and credit assignment, and the exploration-exploitation dilemma that makes RL its own discipline.
Behind every reinforcement learning algorithm is one formal structure: the Markov decision process. It's the precise way of writing down "an agent making a sequence of decisions under uncertainty," and once a problem is framed as an MDP, the whole toolkit of RL applies to it. This post builds the MDP from its parts and explains the one assumption — the Markov property — that makes the entire theory work.
Behind every RL algorithm is one structure: the Markov decision process (states, actions, transitions, reward, discount). The Markov property that makes it tractable, why state design matters, and reward design as RL's subtle, decisive, error-prone part (reward hacking) — the problem RLHF later solves.
The key idea that makes reinforcement learning work is deceptively simple: instead of chasing immediate reward, estimate the long-term value of being in a situation, and act to move toward high-value situations. Value functions are how an agent "thinks ahead," and the Bellman equation is the elegant recursive relationship that makes computing them possible. This is the conceptual core of RL — grasp it and the algorithms become obvious.
The idea that makes RL work: estimate the long-term value of a situation and move toward high-value ones. State-value and Q-functions, the Bellman equation (value = reward + γ × value-of-next) that turns infinite look-ahead into a recursion and solves credit assignment, and planning vs learning.
The elegant idea of value functions becomes a practical algorithm through one trick: learn by comparing your prediction to a slightly-better prediction, and nudge toward the difference. Temporal-difference learning and its famous instance, Q-learning, are how an agent learns the value of its actions purely from experience — no model of the world required. This is the algorithm that first made reinforcement learning work, and its core idea quietly underlies much of the field.
Learn a guess from a better guess: temporal-difference learning and Q-learning nudge value estimates toward reward + γ × value-of-next, model-free, bootstrapping from every step. Off-policy learning, ε-greedy exploration, and why the Q-table can't scale — motivating deep RL.
Reinforcement learning was a tidy theory stuck on toy problems until it was combined with deep learning. Replacing the value table with a neural network let RL handle the enormous, continuous state spaces of real problems — and the result, deep RL, is what beat humans at Atari, Go, and StarCraft, and what trains agents in the real world today. This post is about that fusion: why it's powerful, why it's notoriously unstable, and the tricks that make it work.
Replacing the value table with a neural network let RL handle huge, continuous, pixel-level state spaces — the leap that beat humans at Atari and Go. Why naive deep RL diverges, the tricks that tame it (experience replay, target networks), and why it's powerful, brittle, and sample-hungry.
Value-based methods like Q-learning learn how good each action is, then act greedily. There's a different and often better approach: skip the value estimates and learn the policy itself, directly adjusting the probability of actions to get more reward. Policy gradient methods are how RL handles continuous control and stochastic strategies — and the actor-critic architecture that combines them with value learning is the foundation of the algorithms used to fine-tune today's language models.
Instead of learning values then acting greedily, learn the policy directly: adjust action probabilities to earn more reward. Why this handles continuous and stochastic policies, the high-variance problem, and actor-critic (and PPO) — the algorithm family behind RLHF and LLM alignment.
The single most consequential application of reinforcement learning today isn't a game or a robot — it's the technique that turned raw, unruly language models into helpful assistants. Reinforcement learning from human feedback solved the problem that you can't write a reward function for "be helpful, honest, and harmless," so you learn one from human judgments instead. This post connects everything in the series to the method that made modern AI assistants possible.
The most consequential use of RL today: you can't write a reward for 'be helpful, honest, harmless', so RLHF learns one from human preference comparisons and optimizes with PPO. The three-stage pipeline, the KL constraint against reward hacking, and the series' capstone — plus its limits (sycophancy, bias).
Reinforcement learning is the most seductive paradigm in machine learning: define a goal, let an agent figure out the rest. It's also the hardest to get working and the easiest to misapply. This closing post is the practical judgment the series was building toward — where RL genuinely shines, where a simpler method is the honest choice, and how to recognize which situation you're in. It's the difference between RL as a tool and RL as a trap.
RL is the most seductive ML paradigm and the easiest to misapply. Where it genuinely shines (games, control, alignment), where a simpler method is the honest choice, and how to tell which — the difference between RL as a tool and RL as a trap. Its deepest value: optimizing goals you can judge but not specify.
This series is part of a larger body of work by Pratik Dhanave, an Agentic AI Architect writing about production AI systems, distributed systems, and cloud-native engineering. Explore all course series, browse every post, or find topics via the tag index.