The Big Picture
Convert evaluated failures into persistent procedure fixes by evolving an external harness state (episodic memory + state graph), and you can boost task success by about 10 percentage points without changing the base model or tools.
ON THIS PAGE
Key Findings
Living-Harness turns after-action evaluations into bounded, reusable updates to an external harness that stores experiential memory and workflow rules. The system keeps tools and base context frozen but evolves two artifacts: episodic memory (episodic memory) (trigger, failure pattern, recovery action) and a state graph (state nodes and repair edges). Across dialogue and tool-driven benchmarks, these persistent repairs accumulate across cycles and improve task success, and the evolved harness can be reused by different model backbones via retrieval-only access. external harness
By the Numbers
1Improved average Pass@1 by 10.07 percentage points on τ2-Bench compared to the strongest interactive baseline.
2Improved average Pass@1 by 9.91 percentage points on MultiWOZ-2.4 compared to the strongest interactive baseline.
3Memory and state-graph retrieval use top-k = 3 entries; task-local retries were limited to at most 3 trials per task during evaluation.
Why It Matters
Engineers building multi-turn, tool-using agents: can add persistent procedural fixes without retraining core models. Technical leaders evaluating agent reliability: offers a practical path to reduce repeated failures by evolving a small, auditable state. Researchers tracking agent adaptation: shows a way to convert evaluator signals into durable workflow changes that transfer across model backbones.
Test your agentsValidate against real scenarios
Key Figures
Fig 1: Figure 1: Static vs. living harnesses. A static harness remains fixed after deployment. Living-Harness uses an Evolution-SOP to guide bounded updates to episodic memory and the state graph for future retrieval, while tools and base context remain frozen.
Fig 2: Figure 2: Overview of Living-Harness. In episode n n , the agent interacts with the environment under harness state S ( n ) S^{(n)} , producing trajectory τ n \tau_{n} and evaluation signals y n y_{n} . Guided by the Evolution-SOP ψ d \psi_{d} , these signals are converted into posterior evidence and used to update episodic memory ℛ \mathcal{R} and the state graph G G . The resulting state S ( n + 1 ) S^{(n+1)} is written back and retrieved to guide future episodes, while tools and base context remain frozen.
Fig 3: Figure 3: Component ablation on τ 2 \tau^{2} -Bench: removing the Evolution-SOP causes the largest drop, while memory and state graph ablations show complementary contributions.
Fig 4: Figure 4: Case study of persistent procedural repair. In Cycle 0, Reflexion recognizes that the user should be transferred but repeatedly omits the required transfer_to_human_agents() call. Guided by the Evolution-SOP, Living-Harness converts this evaluated failure into an episodic-memory entry and a state-graph repair edge. In Cycle 1, the repaired harness retrieves the missing procedure, executes the tool call, and completes the task in one attempt.
Ready to evaluate your AI agents?
Learn how ReputAgent helps teams build trustworthy AI through systematic evaluation.
Learn MoreYes, But...
The evolution rules (Evolution-SOP) are domain-specific and must be carefully designed and frozen to avoid uncontrolled changes. Experiments used eight simulated domains and a simulated user in some cases, so behavior in messy, real-world user traffic may differ. Persisting incorrect repairs is possible, so human oversight and safety checks are needed before committing automated updates in production. State Inconsistency
Deep Dive
Living-Harness separates the agent's fixed execution resources (tools and base context) from an evolving procedural state that captures both experience and workflow structure. After each interaction episode, an external evaluator rates the outcome. A run-frozen Evolution-SOP (a domain-level rulebook) converts the completed trajectory plus evaluator signals into two bounded updates: (1) episodic memory records that store trigger conditions, failure patterns, and concrete recovery actions; and (2) a state graph that adds or modifies state nodes, transition rules, and repair edges. On subsequent episodes, the agent retrieves the most relevant memory and graph entries (top-3) and uses them as procedural context so recurring failures can be prevented rather than merely retried. Evaluations on two benchmark suites covering eight domains show roughly ten percentage-point gains in task success (Pass@1) over the strongest interactive baselines, and ablation studies indicate the Evolution-SOP is the most critical component. A concrete case: when the agent repeatedly failed to call a transfer-to-human tool, Living-Harness converted the evaluated failure into a memory entry and a state-graph repair, and the next cycle executed the required tool call and completed the task. The approach improves agent reliability without modifying base models or tools, but requires careful SOP design, monitoring for mistaken persistent repairs, and further validation in real production traffic. Evolution-SOP
Test your agentsValidate against real scenarios
Credibility Assessment:
No affiliations specified and most authors have very low h-indices; arXiv preprint with no citations indicates emerging/limited credibility.