Where Artifact Evolution Works: The Verifier Spectrum

Artifact evolution's track record is strangely lumpy: superhuman results on matrix-multiply schedules and scheduling heuristics, respectable results on general code optimization, and little to show on "write me a better product strategy." The lumpiness isn't mysterious — it traces exactly one variable: how good a verifier the domain affords. This post organizes the application domains along that spectrum, from exact checkers to vibes, and reads off where the loop is trustworthy, where it needs guardrails, and where it quietly becomes fiction.

Post 2 ended on the claim that the evaluator is the system. Take that seriously and the question “where does artifact evolution work?” reduces to “where can you build an honest evaluator?” — which is a question about domains, not about models.

The spectrum, strong end first

Reading the spectrum

Three rules fall out. Search pressure must not exceed verifier strength — the amount of optimization a signal can absorb before it stops meaning anything is finite and ordered exactly as above; thousand-round searches belong on exact and measured verifiers, judge-scored domains tolerate only gentle pressure. The verifier, not the generator, is where engineering effort compounds — a better test suite or a denoised benchmark improves every future search; a cleverer prompt improves one. Verifier quality is a property of the domain you choose, partly under your control — the highest-leverage move in applying artifact evolution is often reshaping the problem until it affords a stronger verifier: extracting a checkable kernel from a vague goal, adding a reference implementation, formalizing the acceptance criterion. The domains where self-evolving AI “mysteriously works” are the domains someone did that reshaping first.

The takeaway: artifact evolution’s successes map one-to-one onto verifier strength — exact checkers (math, formal proofs) at the trustworthy end, timed execution and test suites in the engineerable middle, proxy scores and LLM judges at the fragile end where Goodhart stops being a risk and becomes the expected outcome. Match search pressure to verifier strength, invest in the verifier before the generator, and reshape problems toward checkability — that discipline, not model choice, decides whether the loop produces results or fiction.

Key takeaways

Further reading

Sources & References

the sampled verifier most real systems live on