Building an Eval Harness

We have metrics (post 2) and LLM judges (post 3). Now we assemble them into something you can actually run repeatedly. A harness is the difference between “I evaluated it once, by hand” and “evaluation runs automatically on every change and blocks regressions.” This post is about its anatomy: the dataset, the runner, the scorers, and the report.

The four parts of a harness

Strip away the tooling and every eval harness has the same shape:

1. A dataset — a collection of test cases, each an input (and often an expected output or grading rubric).
2. A runner — code that feeds each input to the system under test and collects the output, handling the messy realities of API calls.
3. Scorers — the metrics and/or judges from the previous posts, applied to each output to produce per-case scores.
4. A report — aggregation and presentation: overall scores, breakdowns by category, and per-case detail for debugging.

Existing frameworks (EleutherAI’s lm-evaluation-harness, OpenAI’s evals, and others) implement exactly this shape; understanding the parts lets you use them well or build a lean one yourself. The value is in the parts working together reproducibly, not in any particular tool.

The dataset is the hard part

Everything downstream is only as good as the test cases. A weak dataset produces confident, meaningless numbers. Good eval datasets share a few properties:

• Representative of real usage. The cases should mirror the distribution of inputs your system actually sees — the common paths and the important tails. A dataset of only easy questions tells you nothing about the hard ones users will send.
• Includes edge cases and known failures. Every bug you find in production should become a permanent test case. This is how the dataset accumulates institutional memory: it grows a immune system against past mistakes, so a fix that regresses is caught immediately.
• Labeled or gradeable. Each case needs a way to be scored — a reference answer, a rubric for a judge, or a functional check. Cases you can’t score aren’t tests.
• Categorized. Tag cases by type (question category, difficulty, feature area) so the report can show where you’re strong and weak, not just an average that hides everything.
• Sized deliberately. Big enough that the score is statistically meaningful and not dominated by a handful of cases; small enough to run cheaply and often. Start with dozens of high-quality cases over thousands of junk ones — a curated set beats a scraped one.

A recurring question is where the cases come from: hand-written by domain experts (highest quality, slowest), sampled and labeled from real production logs (most representative), or synthetically generated by a model (fast, scalable, but risks baking in a model’s blind spots). A blend is typical — seed with expert cases, expand with labeled production traffic, augment with synthetic edge cases you then review.

The golden set

A special, high-value artifact is the golden set: a curated, stable collection of cases with carefully verified correct answers, treated as the authoritative benchmark for your system. It changes rarely and deliberately. Its stability is the point — because it doesn’t move, scores across it are comparable over months, so you can say “we went from 82% to 89% on the golden set since the spring” and mean it. Keep it version-controlled, review changes to it like code, and never edit it casually to make a number look better (that’s tampering with your own scale).

The runner: handling reality

The runner looks trivial — loop over cases, call the model — but production LLM APIs are hostile to naive loops:

• Non-determinism. The same input can yield different outputs. For eval, pin what you can (temperature=0 where the task allows, fixed seeds if available) to reduce run-to-run variance, and where output genuinely varies, run each case multiple times and aggregate rather than trusting a single sample.
• Failures and rate limits. Calls time out, rate-limit, and error. The runner needs retries with backoff, and a policy for cases that still fail (record them as failures — don’t silently drop them, which would inflate your score by removing hard cases).
• Cost and latency. Evaluating thousands of cases across variants gets expensive. Run cases concurrently (within rate limits), cache results keyed by (input, system-version) so re-runs are cheap, and record cost and latency as metrics — they’re part of quality.
• Reproducibility. Record the exact system version, prompt, model, and parameters alongside every result, so a score is always attributable to a specific configuration. An eval result you can’t reproduce or attribute is an anecdote.

The report: aggregate and drill down

Finally, turn per-case scores into decisions. A good report does two things at once:

• Aggregates — overall score and per-category breakdowns, so you see both the headline and where the strengths and weaknesses are. Averages alone are dangerous: a flat 85% can hide a category sitting at 40%.
• Enables drill-down — the ability to open any individual case and see the input, output, score, and (if using a judge) the grading rationale. This is where debugging happens: a number tells you that something regressed; the per-case detail tells you what and why.

The most useful report of all is a comparison: candidate vs. baseline, showing not just the score delta but which specific cases flipped from pass to fail and back. This directly answers the question that drives every change — “did this help, and what did it cost me?” — and surfaces the silent regressions that averages hide.

Wire it into CI

The final move that makes a harness pay off: run it automatically. Just as unit tests run on every commit, your eval suite should run on every meaningful change to prompts, models, or system code — ideally in CI, gating deploys on a minimum score and flagging regressions before they merge. (We’ll return to CI gating in the production post.) A harness you run by hand once a month is a nice report; a harness wired into CI is a safety net that makes fearless iteration possible.

Key takeaways

• Every eval harness has four parts: a dataset of test cases, a runner that executes the system, scorers (metrics/judges), and a report — frameworks like lm-evaluation-harness and OpenAI evals implement this shape.
• The dataset is the hard part: it must be representative, include edge cases and past production bugs, be labeled/gradeable, categorized, and deliberately sized — a small curated set beats a large scraped one.
• Maintain a golden set — a stable, verified, version-controlled benchmark — so scores are comparable over months; never edit it casually to flatter a number.
• The runner must handle reality: non-determinism (pin temperature/seeds, sample multiple times), retries/rate-limits, concurrency, caching, cost/latency as metrics, and full reproducibility (record model, prompt, params per result).
• The report must aggregate and drill down — category breakdowns plus per-case detail with grading rationale — and the most valuable view is a candidate-vs-baseline comparison showing which cases flipped.
• Wire the harness into CI so evals run on every change and gate deploys — that’s what turns evaluation from a periodic report into a regression-catching safety net.

Further reading

• OpenAI Evals — framework and registry
• EleutherAI — lm-evaluation-harness