The Big Picture
Diagnosing failures from execution traces and applying small, scoped fixes to the runtime around models (the harness) can improve AI agent success rates by 15–50% while avoiding many regressions.
ON THIS PAGE
The Evidence
Fine-grained, trace-grounded diagnosis that links failed runtime steps to specific harness mechanisms lets you target repairs where they matter most. Applying scoped fixes (not wholesale prompt rewrites) and validating them on held-out tasks raised success rates across four benchmarks. These repairs touched layers beyond prompts—like tool interfaces, execution control, logging, and verification—and outperformed both human-designed harnesses and prompt-only self-evolution methods.
Data Highlights
1Held-out test improvements: SWE-Bench 45 → 57 (26.7%); Terminal-Bench 6 → 9 (50.0%); GAIA 26 → 37 (42.3%); AppWorld 33 → 38 (15.2%).
2Beats top human harnesses on every benchmark (e.g., 57 vs 48 on SWE-Bench, 9 vs 7 on Terminal-Bench, 37 vs 35 on GAIA, 38 vs 37 on AppWorld).
3Outperforms self-evolution baselines by margins of 5–10 tasks on SWE-Bench, 2–3 on Terminal-Bench, 5–10 on GAIA, and 2–4 on AppWorld.
What This Means
Platform engineers and reliability teams who run AI agents—because targeted harness fixes are often cheaper and faster than retraining models and reduce repeat failures. Tool builders and researchers designing agent pipelines—because trace-grounded diagnosis reveals where to change code, prompts, or adapters rather than guessing from outcomes alone.
Not sure where to start?Get personalized recommendations
Key Figures
Fig 1: Figure 1 : Overview of HarnessFix .
Fig 2: Figure 2 : Illustrative example for the AppWorld completion-guard harness flaw.
Ready to evaluate your AI agents?
Learn how ReputAgent helps teams build trustworthy AI through systematic evaluation.
Learn MoreYes, But...
Results come from four benchmarks (software repo edits, terminal workflows, open-ended research, and app automation); other domains may show different flaw patterns. The method depends on rich, structured execution traces and a clear mapping from runtime steps to harness artifacts; sparse or unaligned logs will weaken diagnosis. Repairs are generated and validated by model-driven agents, so human review or conservative rollout remains important to catch subtle correctness or safety issues. structured execution traces
Methodology & More
HarnessFix turns failed agent executions into actionable repairs by first normalizing heterogeneous traces into a step-level representation (HTIR) that links inputs, tool calls, model calls, and outcomes. A diagnosis agent attributes failures to specific trace steps and to harness layers (execution, tool interface, context, lifecycle, observability, verification, governance). A repair agent maps recurring flaw patterns to a small set of scoped repair operators, generates candidate edits to prompts, adapters, or harness code, and a validation agent accepts patches only when they reduce the target flaw without exceeding regression bounds on held-out tasks.
Across four benchmark families, this trace-guided, diagnosis-driven process improved held-out task success rates by roughly 15–50% and beat both prompt-only self-evolution and the strongest human-designed harnesses. Accepted fixes included concrete changes such as preserving file attachments before reasoning, blocking dangerous terminal commands, adding final-readonly API verification, and recording timeouts as unsolved events instead of runner errors. The key practical takeaway: fixing the runtime around models (logs, tool plumbing, lifecycle checks, and verification rules) often yields larger and safer gains than changing the model-facing prompt alone, and should be part of any production agent improvement workflow.
Avoid common pitfallsLearn what failures to watch for
Credibility Assessment:
Multiple authors but no affiliations or h-index info, arXiv preprint — emerging/limited information.