Key Takeaway
AutoSaddler automatically diagnoses long-run agent failures from execution traces and applies targeted, test-verified fixes to the surrounding system, boosting task success by about 9–10 percentage points and producing durable, generalizable updates.
ON THIS PAGE
Key Findings
Harnesses—the prompts, tool settings, and runtime logic that surround an AI model—can be automatically improved by treating them like code and iteratively patching failures seen in execution traces. AutoSaddler runs a loop of diagnosis, structured patch generation, batch verification, and history-aware selection to prefer fixes that generalize beyond the observed failures. Across three diverse benchmarks, these durable updates raised multi-step task success substantially compared to base harnesses and beat the best automated baselines. The approach depends on deep debugging of traces, targeted (not ad-hoc) edits, and explicit checks that changes help on held-out examples. In particular, careful diagnosis of failures guides effective interventions.
Not sure where to start?Get personalized recommendations
Data Highlights
19.0, 9.6, and 10.0 percentage-point gains in task success on GAIA2, SWE-Bench Pro, and Terminal-Bench 2.0 respectively versus the corresponding base harnesses.
2AutoSaddler beat the strongest automated baseline by 7.4 points (GAIA2), 4.4 points (SWE-Bench Pro), and 6.7 points (Terminal-Bench 2.0).
3A two-phase optimization (exploration then refinement) produced a cumulative 24.6 percentage-point improvement during the optimization trajectory.
What This Means
Engineers building multi-step or long-horizon AI agents: use this approach to reduce manual harness tuning and get more reliable end-to-end behavior. Technical leads and SREs responsible for production agent reliability: AutoSaddler-style tooling can cut down repeated debugging and produce fixes that generalize across tasks. Researchers tracking agent evaluation: this gives a practical path to automate and measure how harness changes affect real agent runs.
Key Figures
Fig 2: Figure 2 : Overview of AutoSaddler . The iterative optimization loop: the current harness is tested on a mini-batch, diagnosed and patched across harness components, verified for improvement, and then reflected upon to extract lessons into the EvoDAG, which guides the evolution of the next harness.
Fig 7: Figure 7 : Case study on a calendar task with ambiguous cancellations, where AutoSaddler’s in-depth diagnosis accurately identifies the true root cause via deep debugging, unlike shallow diagnosis.
Fig 8: Figure 8 : Case study on a file-management task that requires moving PNG files into a sims3 folder under benchmarks . Shallow diagnosis misattributes the failure to a relative-path error, while AutoSaddler verifies the filesystem and locates the true cause: an unverified parent directory.
Fig 9: Figure 9 : Case study on a cab-booking task with two candidate rides on the same day. Shallow diagnosis overlooks the second matching ride and proposes an off-target fix, while AutoSaddler inspects the full ride history and identifies the missed candidate.
Ready to evaluate your AI agents?
Learn how ReputAgent helps teams build trustworthy AI through systematic evaluation.
Learn MoreYes, But...
AutoSaddler operates offline on batches of tasks, so it requires representative training and validation sets to avoid missing real-world failure modes. The system assumes the harness can be expressed and patched in structured ways; truly novel tools or drastic architectural changes may need manual engineering. Experiments are on three public benchmarks and specific base harnesses, so results may vary for different domains or very different agent designs. structured ways to express and patch the harness are important for success.
Full Analysis
AI agents can look smart on some subtasks and fail on others, and the surrounding harness (prompts, tools, runtime glue) often holds the key to consistent performance. AutoSaddler treats harness optimization as an offline learning problem: run the current harness on mini-batches of tasks, diagnose failed multi-step traces in depth, propose structured patches (think code edits to prompts, tool descriptions, or control hooks), verify whether the patch improves the mini-batch, then validate on held-out data. Each accepted update is recorded in a history graph called EvoDAG, which the system uses to propose subsequent candidates and avoid regressions. The pipeline emphasizes three ideas: deep diagnosis (inspect full execution traces rather than surface signals), diagnose failed multi-step traces in depth, and generalization-aware selection (only accept updates that show improvement on validation tasks). On three diverse benchmarks covering personal assistant, software engineering, and terminal tasks, this yields consistent and sizable gains—around 9–10 percentage points versus base harnesses and clear wins over automated baselines. Practically, this approach can reduce manual tuning work and produce durable fixes, though it requires representative task data and a harness design amenable to structured patches.
Explore evaluation patternsSee how to apply these findings
Credibility Assessment:
Several authors have moderate h-indices in the 10–14 range indicating established researchers, but venue is arXiv and affiliations aren’t specified — solid/recognized but not top-tier.