Model Evolution I: Learning Without Gold Answers

The deepest locus of self-evolution is the weights — and the interesting frontier there is not supervised fine-tuning on human labels but the family of methods where the model improves from signals it generates or uncovers itself: pseudo-labels from its own consensus, rewards from self-play, curricula it proposes to itself. This post organizes that family by one question — where does the training signal come from when no gold answer exists — and traces the shared failure mode every branch inherits: a model learning from itself can amplify its knowledge, and can just as efficiently amplify its errors.

Post 4 ended at the harness ceiling: scaffolding elicits frozen ability but can’t exceed it. Crossing the ceiling means changing weights — and the self-evolution question is whether weights can improve without the expensive ingredient, human-labeled answers. The answer is a qualified yes, with the qualification doing a lot of work.

The signal taxonomy

Every labeled-data-free training method answers “what do we train toward?” one of four ways:

The shared failure mode: drift without an anchor

All four branches train the model on data shaped by the model, and that circularity has one universal consequence: errors compound unless something external breaks the cycle. The mechanisms differ by branch — consensus launders bias, self-play escapes into private equilibria, verifier-reward internalizes exploits, environment signals reinforce whatever got rewarded including the accidents — but the discipline is the same, and it’s the closest thing this field has to a safety checklist: keep an anchor (reference data, an executor, a frozen baseline model, periodic human-verified batches) that the loop cannot modify; evaluate on held-out, never-trained-on measures watching for the signature of drift (self-scored metrics up, external metrics flat or down); and gate weight updates like production deploys — staged, compared against the frozen baseline, reversible. The branches that respect the anchor have produced real capability gains; the ones that don’t produce models that are extremely good at agreeing with themselves — the weight-level rhyme of everything post 3 said about verifier pressure, at higher stakes, because the exploit ends up in the weights.

The takeaway: label-free model evolution is a four-branch family sorted by signal source — consensus (sharpens existing knowledge, launders shared bias), self-play (manufactures a scaling curriculum, risks private equilibria), verifier reward (sturdiest, inherits and internalizes verifier weaknesses), and environment signal (honest but slow, noisy, and governance-heavy). The circularity is common to all: the model trains on model-shaped data, so drift is the default — and external anchors, held-out evaluation, and deploy-grade gating on weight updates are what separate self-improvement from self-reinforcement.

Key takeaways

Further reading

Sources & References