The Big Picture
Reusing past runs smarter (by tying evidence to specific functions and recalibrating whether that evidence still applies) makes automated harness updates steadier and yields about 16% better reasoning and 14% better agent-task performance under limited evaluation budgets.
ON THIS PAGE
Key Findings
Historical trial logs often mislead automated harness updates because evidence becomes stale or lacks guidance on which component to change. Turning each trial into function-level evidencefunction-level evidence, reweighting that evidence for the current harness state, and distilling it into explicit guidance for where to edit leads to smoother, less oscillatory evolution. With this approach, harness search finds higher-performing configurations faster and wastes fewer evaluation rounds.
Test your agentsValidate against real scenarios
By the Numbers
1Average performance gain of 16.2% on domain reasoning benchmarks (USPTO, Symptom2Disease, Law) versus evaluated baselines.
2Average performance gain of 14.2% on agentic tasks (terminal interaction and code repair) versus evaluated baselines.
3Evolution was run under constrained budgets (20 iterations for domain reasoning, 10 iterations for agentic tasks) and produced smoother, more stable improvement trajectories.
Why It Matters
Engineers building AI agents and teams automating agent toolchains benefit because the method reduces wasted evaluation runs and yields more reliable harness updates. Technical leads and researchers evaluating agent reliability can use the approach to get steadier improvements when tuning multi-component execution frameworks under tight budgets. For practitioners, optimizing toolchains can be aided by applying toolchains patterns to streamline updates.
Key Figures
Fig 1: Figure 1: (a) Full-history and compressed-history reuse (Meta-harness, Claude Opus 4.6, 1M-token context window) both exhibit non-stationary evolution across the S2D, USPTO, and Law datasets, with accuracy repeatedly regressing and recovering across iterations despite accumulating historical experience. (b) Two limitations in harness self-evolution: What is still valid? and Where to evolve next?
Fig 2: Figure 2: Detailed overview of the DREvo framework, which anchors historical evidence at the function level, recalibrates its validity under the current harness state, and distills it into explicit guidance for harness self-evolution under limited budgets.
Fig 3: Figure 3: Validation set accuracy (Acc.) and context (Ctx.) size during harness self-evolution for DREvo and Meta-harness (compressed-history) on domain reasoning tasks.
Fig 4: Figure 4: Performance changes under controlled component state drift during harness evolution. Drift perturbs the target component while keeping historical evidence fixed. Blind Reuse applies evidence without state calibration.
Ready to evaluate your AI agents?
Learn how ReputAgent helps teams build trustworthy AI through systematic evaluation.
Learn MoreYes, But...
Results were obtained with specific model choices for proposer and evaluators, so gains may vary with different language models or tool stacks. Benchmarks cover domain reasoning and agentic tasks but do not guarantee the same improvements on all real-world applications or much longer evolution horizons. The method focuses on per-function evidence and role-conditioned updates—cross-component calibration and adaptive long-run role policies remain areas for future work. See cross-component calibration for related considerations.
Deep Dive
Harness updates driven by accumulated trial history often oscillate because past evidence can become invalid as the harness changes, and historical logs rarely say which component to change next. The approach here converts raw trial logs into function-anchored evidence units (so each piece of feedback points to a specific function or component), recalibrates each unit’s reliability for the current harness state (downweighting or rejecting stale evidence), and then distills relevant, calibrated evidence into an explicit, role-conditioned search intent that tells the proposer where and how to modify the harness. Evaluation used three domain reasoning datasets (chemical reaction prediction, medical diagnosis, legal reasoning) and two agentic suites (terminal interaction and code repair). Evolution runs used limited budgets (20 iterations for domain tasks, 10 for agentic tasks) and compared against non-evolution baselines and recent evolution methods. The recalibrated, function-level approach produced smoother evolution trajectories with fewer performance regressions and delivered average gains of 16.2% on domain reasoning and 14.2% on agentic tasks. Practically, this means fewer wasted test runs and more reliable automated harness tuning, making it easier to automate improvements to an agent’s execution framework under tight resource limits. See domain reasoning datasets for related examples and consider LLM-as-Judge Pattern to further refine evaluation or emergence-aware monitoring pattern for ongoing monitoring.
Test your agentsValidate against real scenarios
Credibility Assessment:
Affiliated with Southeast University (recognized institution) though authors have low h-indices and it’s an arXiv preprint; solid but not top-tier.