Evaluating Coding Agents — SWE-bench and Measuring Real Capability

Posts 1–6 built how coding agents work; this post is how we measure them. Evaluation matters because capability claims are everywhere and mostly unverified, and because code’s verifiability (post 1) makes rigorous evaluation uniquely possible here. It connects to the blog’s AI evaluation coverage, specialized for coding agents.

Why coding agents are especially evaluable

Coding agents are, happily, one of the easiest AI systems to evaluate rigorously — for the same reason they work at all: code is verifiable (post 1). This makes evaluation objective in a way most AI eval isn’t:

• Pass/fail by tests, not opinion. You can evaluate a coding agent by giving it a real task and checking whether its solution passes the tests — an objective, automatic verdict, not a fuzzy “does this look good?” (the problem that plagues LLM evaluation generally, the AI evaluation series). Did the agent resolve the issue, verified by tests? Yes or no. This objectivity is a gift — coding-agent eval can be rigorous where most AI eval is subjective.
• Real tasks are available. The world is full of real software tasks with known-correct solutions and tests — GitHub issues, bug fixes, feature requests, each with a verifying test suite. So you can evaluate agents on actual engineering tasks, not toy problems, with ground-truth verification. Real, verifiable tasks are abundant.

This is why coding-agent benchmarks became rigorous and influential: the domain affords objective, realistic evaluation. The headline benchmark, SWE-bench, embodies this — it gives agents real GitHub issues from real open-source projects and checks whether the agent’s fix makes the project’s real tests pass. It measures the thing that actually matters (can the agent resolve a genuine issue, test-verified?), which is why “% of SWE-bench resolved” became the standard coding-agent capability metric and a driver of rapid, measurable progress. The ability to put a real, test-verified number on coding-agent capability is itself a reason the field advanced so fast — progress you can measure is progress you can optimize.

What’s measured, and the pitfalls

Coding-agent evaluation measures several things, each with pitfalls to know:

• Task resolution rate (the headline). What fraction of real tasks the agent resolves (solution passes the verifying tests) — the core capability measure (SWE-bench-style). The pitfall: a single headline number hides which tasks (easy vs hard, which projects, which languages), so compare like-for-like and look at the task distribution, not just the percentage.
• Correctness beyond tests. Passing tests is necessary but not sufficient — a solution can pass tests yet be a bad fix (wrong approach, poor quality, breaks something untested). So evaluation ideally also checks quality (is it a good solution?), which is harder and more subjective (back toward the fuzzy-eval problem). “Passes tests” is a strong but incomplete signal.
• Efficiency and cost. How many steps, how much compute/token cost, how long the agent took to resolve a task — because an agent that resolves tasks but at huge cost or after hundreds of flailing steps is less useful than the headline rate suggests. Capability per cost matters, not just capability.
• The pitfalls that inflate numbers — the recurring AI-eval dangers, sharpened for coding:
• Benchmark contamination — if the benchmark’s tasks (and solutions) were in the model’s training data, high scores reflect memorization, not capability (the leakage problem from the ML and IR series). Public benchmarks age into training sets, so contamination is a serious, growing concern — a number on a contaminated benchmark is not real capability.
• Overfitting to the benchmark — optimizing an agent to score well on SWE-bench specifically may not generalize to your real tasks (Goodhart’s law — the ML eval point). The benchmark is a proxy, not the goal.
• Weak test suites — if a task’s verifying tests are weak, “passing” them doesn’t mean the fix is correct (post 6’s point — verification is only as good as the tests). Benchmark quality depends on test quality.
• Cherry-picked demos — vendor demos show the wins, not the failure rate (the AI-FDE series’s “the demo is a trap”). Real capability is the resolution rate on a representative, uncontaminated task set, not a highlight reel.

So reading coding-agent capability claims requires the eval skepticism the blog preaches throughout: what tasks, verified how, contaminated or not, at what cost, and does it generalize to my work?

Evaluating for your use — and the honest picture

Beyond benchmark numbers, the practical question is how good is a coding agent for your work, and evaluation there is its own discipline:

• Your tasks are the real benchmark. SWE-bench measures general capability, but what matters is whether the agent resolves your tasks in your codebase — so evaluate on your real tasks (the LLMOps and AI eval “your eval set is the asset” lesson, for coding). Build a set of representative real tasks from your work and measure the agent on those, with your tests as the verifier.
• Measure resolution and quality and review burden. For your use, what matters is: does it resolve the task (tests pass), is the solution good (you’d merge it), and how much review did it take (an agent whose output always needs heavy rework is less valuable than its raw resolution rate suggests). The real metric is useful-work-per-review-effort, not just pass rate.
• The honest capability picture. Coding agents have gone from resolving a small fraction of real issues to a large one on benchmarks — genuine, rapid, measured progress — and they’re genuinely useful on well-scoped, well-tested tasks. But benchmark numbers overstate real-world reliability (contamination, overfitting, weak tests, the gap between “passes tests” and “good fix”), and agents still struggle on the hard, ambiguous, weakly-verified tasks (post 8). The honest read: impressive and improving fast, genuinely useful within their reliable zone, but not as universally capable as headline numbers suggest — which is exactly why evaluation (rigorous, on-your-tasks, skeptical of benchmarks) is how you know what to actually rely on.

The through-line: coding agents are uniquely evaluable because code is verifiable — you can measure task resolution by tests (objective pass/fail, not fuzzy opinion) on real tasks (GitHub issues with real tests), which is why SWE-bench (“% of real, test-verified issues resolved”) became the standard metric and drove measurable progress. But reading the numbers needs skepticism: beware contamination (benchmark tasks in training = memorization not capability — growing concern), overfitting to the benchmark (Goodhart — may not generalize to your work), weak test suites (passing ≠ correct), and cherry-picked demos; and measure cost/efficiency, not just resolution. For your use, your real tasks are the real benchmark — evaluate resolution and solution quality and review burden (useful-work-per-review-effort). The honest picture: impressive, improving fast, genuinely useful within their reliable zone, but less universally capable than headline numbers suggest — so rigorous, on-your-tasks, benchmark-skeptical evaluation is how you know what to rely on.

Key takeaways

• Coding agents are uniquely evaluable because code is verifiable (post 1): evaluate by task resolution — does the agent’s solution pass the verifying tests? — an objective pass/fail verdict (unlike the fuzzy “does this look good?” that plagues general LLM eval), on real tasks (abundant GitHub issues with real test suites).
• SWE-bench embodies this — real GitHub issues checked by the projects’ real tests — so ”% of SWE-bench resolved” became the standard capability metric and drove rapid, measurable progress (measurable progress is optimizable progress).
• Reading the numbers needs skepticism about pitfalls that inflate them: benchmark contamination (tasks/solutions in training = memorization, not capability — a serious, growing concern as public benchmarks age into training sets), overfitting to the benchmark (Goodhart — may not generalize to your tasks), weak test suites (passing ≠ correct fix — post 6), and cherry-picked demos (wins shown, failure rate hidden); also measure efficiency/cost (resolution at huge cost or after hundreds of flailing steps is worth less than the headline).
• “Passes tests” is necessary but not sufficient — a solution can pass tests yet be a bad fix (wrong approach, poor quality, breaks untested things), so quality (would you merge it?) matters beyond the pass signal.
• For your use, your real tasks are the real benchmark (build a representative set from your work, verify with your tests — “your eval set is the asset”), measuring resolution and quality and review burden (useful-work-per-review-effort); the honest picture is impressive, improving fast, genuinely useful within their reliable zone (well-scoped, well-tested tasks), but less universally capable than headline numbers suggest — so rigorous, on-your-tasks, benchmark-skeptical evaluation is how you know what to actually rely on.

Further reading

• Benchmark (computing) — measuring capability objectively
• Software testing — the verification that makes coding-agent eval objective