Offline evaluation tells you whether a change is safe to ship; online evaluation tells you whether it actually worked. No matter how good your eval set is, production always surprises you — real users send inputs you never imagined, and the gap between your offline scores and real-world quality is where LLM apps quietly fail. This post is about measuring quality on live traffic and turning user behavior into the feedback that makes the system improve.
Offline evals say a change is safe to ship; online evals say whether it worked. The intrinsic offline–online gap, deploying to measure (canary, A/B, shadow), capturing explicit and implicit feedback, and the pipeline that turns production behavior into continuous improvement.
A recommender that scores well offline can flop in production, and a metric that looks like success can quietly harm the product. Evaluating recommenders is genuinely hard: offline metrics only approximate real behavior, the only ground truth is a live A/B test, and the very act of recommending shapes the data you learn from next. This post covers offline metrics, online testing, the gap between them, and the feedback loops that make evaluation a moving target.
A recommender that scores well offline can flop in production, and a metric that looks like success can quietly harm the product. Evaluation is genuinely hard: offline metrics only approximate real behavior, the only ground truth is a live A/B test, and the very act of recommending shapes the data you learn from next. Offline metrics, online testing, the offline-online gap, and feedback loops.
The most reliable way to establish causation is almost embarrassingly simple: flip a coin to decide who gets the treatment. Randomization is the one technique that defeats confounding completely — including confounders you never thought of — which is why the randomized controlled trial is the gold standard of causal evidence and why the A/B test is the most powerful tool an engineer has. This post explains why randomization works its magic, and the many ways real experiments still go wrong.
The most reliable way to establish causation is almost embarrassingly simple: randomize. Why randomization defeats all confounders at once (even unknown ones), why the A/B test is a genuine causal tool, and the many ways real experiments still go wrong (peeking, power, interference, broken randomization).