The Big Picture
Co-evolving an inspectable evaluation metric with an agent lets the agent self-improve without a preexisting grader, recovering about 88–110% of the lift a true metric would give—provided you anchor the search with a tiny, locked reference set.
ON THIS PAGE
The Evidence
Evolving evaluation metrics as compositions of small, focused detectors (e.g., parse checks, execution tests, narrow LLM judges) produces an inspectable grader that can guide agent self-improvement. A tiny anchored development set (10 examples) is the only supervised signal and is essential: without it the metric loop optimizes noise. Across code generation, text-to-SQL, and report-writing tasks the co-evolved metric plus skill loop (called Double Ratchet) recovered most of the performance that a ground-truth grader would have enabled, and it detected and repaired metric gaming on a realistic report task. Agentic RAG Pattern
Not sure where to start?Get personalized recommendations
Data Highlights
1Evolved metric recovered 88–110% of the supervised (oracle or rubric-driven) performance lift across code, SQL, and report tasks.
2A tiny anchored dev set of just 10 examples functions as the only supervised signal; ablations show these anchors are the primary safety guard against collapse.
3Held-out agreement on report selection was stable around 0.87 (e.g., 0.871 ± 0.063 with 4-item subsamples; 0.882 ± 0.051 with 6-item subsamples), indicating robust transfer from small anchors.
What This Means
Engineers building self-improving agent systems who want to remove or reduce dependence on manual graders: the method gives an inspectable, automatable grader that can be audited and repaired. Technical leaders and product owners evaluating agent reliability can use the anchored-evolution approach to get most of the benefit of supervised evaluation without labeling large numbers of examples. Multi-Agent Software Development
Key Figures
Fig 1: Figure 1: Double Ratchet architecture. The metric loop (left) evolves an expression over drawback detectors; the skill loop (right) evolves the agent’s skill bank, its training graded by the evolved metric during co-evolution. Anchors (bottom) are never trained on; an independent judge audits final outputs on reference-free tasks.
Fig 2: Figure 2: Metric-loop selection objective per round (mean ± \pm std, three seeds; first 70 of 100 rounds, flat afterwards). Chips: peak held-out anchor agreement versus the bare LLM judge op, where ground truth exists to score one.
Fig 3: Figure 3: Held-out learning curves: the reference skill loop (orange, trained on ground truth or the best-available rubric) versus Double Ratchet (blue, trained on its own evolved metric). Mean ± \pm std over three seeds; dotted line = frozen baseline; exact scores in the appendix.
Ready to evaluate your AI agents?
Learn how ReputAgent helps teams build trustworthy AI through systematic evaluation.
Learn MoreConsiderations
The approach does not create ground truth; it expands detector coverage but depends critically on anchor quality and quantity. Some task regimes (e.g., parts of the text-to-SQL experiments) showed weaker transfer, so detectability of failures matters. Experiments used three random seeds and modest held-out sets, so absolute performance numbers have sampling noise and should be validated in each deployment. Context Drift
Methodology & More
Treat the grader as an evolving object: compose a metric from small, deterministic drawback detectors that each check one failure class (static checks, execution tests, or narrow LLM questions). Grow the detector pool by clustering observed failures, gate candidate detectors at birth, score them by marginal contribution, and retire ones that stop helping. Combine detector outputs with logical operators into an expression that is the metric; the root always includes a basic sanity check. Deterministic detectors are chosen to fail differently than the evaluated language model, reducing shared blind spots with the solver being tuned. Run the metric-loop under a three-split protocol: a large unlabeled training split where candidate metrics disagree, a tiny anchored dev split of ten tasks that provides soft pass/fail labels and is the only supervised signal, and a locked test split that no loop ever reads. Co-evolve the metric with a skill-evolution loop (Double Ratchet): the metric grades the skill loop’s training tasks, while an independent locked test (or a stronger LLM judge on reference-free tasks) audits final outputs. In experiments on code generation, enterprise text-to-SQL, and realistic report writing, the co-evolved metric recovered most of the supervised lift (88–110%), produced inspectable compositions that outperformed single LLM judges, and detected Goodhart failures that were repaired by adding detectors. The main safety lesson: anchor discipline (a tiny, locked reference set and an outside auditor) is the critical ingredient that prevents the grader from collapsing into vacuous signals. Chain of Thought Pattern Spiraling Hallucination Loops
Avoid common pitfallsLearn what failures to watch for
Credibility Assessment:
Authors affiliated with Nanyang Technological University (recognized institution) though individual h-indexes are modest — solid but not top-tier signals.