Why Evaluation Is the Bottleneck

Building something with an LLM is easy for a weekend and hard for a year. The wall almost everyone hits is not the model, the prompt, or the framework — it is knowing whether a change made things better or worse. Without a way to measure quality, every improvement is a guess and every deploy is a gamble. Evaluation is the discipline that turns "it seems better" into "it is better, by this much," and it is the real bottleneck in shipping AI systems.

By Pratik Dhanave · August 13, 2026 · 6 min read

#AI Evaluation #AI Engineering #LLMs #Machine Learning #Evaluation

There is a familiar arc to AI projects. A demo comes together in an afternoon and feels magical. Then you try to make it reliably good, and progress grinds to a halt. You tweak the prompt and it fixes one case while breaking two you didn’t notice. You swap models and can’t tell if it helped. You ship, and users find failures you never saw. The thing missing is not capability — it’s measurement. This series is about building that measurement discipline. This first post makes the case for why it matters more than almost anything else you’ll do.

The problem: LLM output is hard to judge

Traditional software has a comforting property: given an input, there’s a correct output, and a test asserts you got it. add(2, 2) must return 4. LLM systems break this. Ask a model to summarize a document and there are thousands of acceptable summaries and no single “correct” string. The output is open-ended, non-deterministic, and quality is a matter of degree, not a boolean.

This wrecks the usual feedback loops. You can’t write assert output == expected when there’s no single expected. You can’t eyeball your way to confidence when the space of inputs is huge and failures are rare and subtle. And you especially can’t trust your gut, because the same fluent, confident prose that makes LLMs impressive also makes bad answers look good. Fluency is not correctness, but our brains conflate them. Evaluation exists to break that spell — to give you a signal about quality that isn’t fooled by tone.

Why “it seems better” is a trap

Without measurement, LLM development degrades into vibes. You make a change, try a few prompts you happen to remember, and if those look good you ship. This fails in three predictable ways:

Regressions hide. Your handful of test prompts is a tiny, biased sample. A prompt change that fixes your favorite examples routinely breaks whole categories you didn’t check. You only learn about them from users — the most expensive possible place to discover a bug.
You can’t compare options. A new model, a cheaper model, a different retrieval strategy, a reworded system prompt — each is a fork in the road, and without a number you’re choosing blind. “It feels a bit sharper” is not a basis for a decision that affects cost, latency, and every user.
Progress isn’t cumulative. With no baseline to beat, you can’t tell whether you’re climbing or walking in circles. Teams thrash for months, each change undoing the last, because nothing is anchored to a measurement.

The cure is the same one that transformed the rest of software engineering: make the invisible measurable, then optimize the measurement.

Evaluation-driven development

The most productive teams treat evals the way disciplined engineers treat tests: as the thing you build first and run constantly. The loop looks like this:

Define what “good” means for your task, concretely enough to measure — accuracy on a labeled set, faithfulness to sources, format compliance, refusal of unsafe requests.
Build an eval set — a collection of representative inputs paired with a way to score the outputs (references, rubrics, or a judge).
Establish a baseline — run your current system and record the score. This number is now the thing to beat.
Change one thing — prompt, model, retrieval, parameters — and re-run the eval.
Keep the change only if the score improves, and watch that you didn’t trade a gain in one category for a loss in another.

This is evaluation-driven development, and it converts the terrifying open-endedness of LLM work into ordinary engineering. Every change gets a verdict. Every deploy has evidence behind it. The eval set becomes the institutional memory of what “good” means, surviving prompt rewrites, model upgrades, and team turnover.

Offline and online: two halves of the loop

Evaluation lives in two places, and you need both.

Offline evaluation happens before you ship: you run a fixed eval set against a candidate system in a controlled way and get a score you can compare against the baseline. It’s fast, cheap, repeatable, and safe — you can try a hundred variations without touching a single user. Offline eval is where you develop. Its weakness is that a fixed dataset can drift out of sync with what real users actually do.

Online evaluation happens in production, on live traffic: A/B tests, guardrail metrics, user feedback signals, and monitoring for quality drift. It measures what actually matters — real behavior on real inputs — but it’s slower, riskier (a bad variant reaches users), and noisier. Online eval is where you confirm.

The healthy pattern is a pipeline: iterate rapidly offline to find promising changes cheaply, then validate the winners online before rolling them out fully. Neither half is sufficient alone. Offline without online optimizes a proxy that may have drifted from reality; online without offline means every experiment is expensive and every idea reaches users unfiltered.

Evaluation is a competitive advantage

It’s tempting to treat evals as overhead — the flossing of AI engineering, virtuous but tedious. That undersells it. In a world where everyone has access to the same frontier models, the differentiator is not the model but how well you can measure and improve your system around it. A team with a strong eval suite iterates faster, ships with confidence, catches regressions before users do, and can safely adopt new models the day they launch (just re-run the evals). A team without one is permanently guessing, slow, and fragile.

That’s why this series treats evaluation as a first-class engineering discipline with its own concepts, tools, and failure modes — not a checkbox. The rest of the series builds it up: what to measure, how to use models as judges, how to construct a harness, how benchmarks are designed and gamed, how humans fit in, and how to close the loop in production. Get this right and the weekend-demo-to-reliable-product wall stops being a wall.

Key takeaways

LLM output is open-ended, non-deterministic, and quality is a matter of degree — there’s rarely a single correct string — which breaks the assert output == expected feedback loops that traditional software relies on.
Without measurement, development degrades into vibes: regressions hide until users find them, you can’t objectively compare models or prompts, and progress isn’t cumulative because nothing is anchored to a baseline.
Fluent, confident prose makes bad answers look good; evaluation exists to give a quality signal that isn’t fooled by tone.
Evaluation-driven development — define “good,” build an eval set, set a baseline, change one thing, keep it only if the score improves — turns open-ended LLM work into ordinary, cumulative engineering.
You need both offline (fast, cheap, safe, pre-ship) and online (real traffic, real behavior, post-ship) evaluation, arranged as a pipeline: iterate offline, validate online.
With everyone using the same frontier models, the ability to measure and improve your system is the real competitive advantage — strong evals mean faster iteration, safer deploys, and instant adoption of new models.