In Brief
AgentLocate reliably identifies which agent and the earliest decisive step caused a multi-agent system to fail by using an LLM to hypothesize, multiple LLM-based evaluators to verify, and iterative fine-tuning to improve accuracy.
ON THIS PAGE
Key Findings
A verify-and-refine workflow that starts with an LLM Judge hypothesis, checks it with multiple diverse Evaluators, and then fine-tunes the Judge from evaluator feedback locates failure sources more reliably than previous one-shot or forensics approaches. On representative benchmarks, the method substantially improves agent-level localization while remaining efficient in prompt tokens and run time. Relying on multiple evaluators and confidence-weighted voting reduces brittleness from single-run judgments, and iterative adaptation further raises fidelity on long, tool-mediated trajectories. Hierarchical Multi-Agent Pattern
Explore evaluation patternsSee how to apply these findings
Data Highlights
1AgentLocate achieved 69.05% agent-level accuracy on the Algorithm-Generated subset, compared to poisoning-forensics baselines that scored below 34%
2Evaluation used three independent evaluators with diverse prompt styles (base, concise, evidence) for each Judge hypothesis
3Benchmarks: Who&When split (50% train, 10% val, 40% test); Aegis-Bench used 600 test samples and 8,933 training samples
Implications
Engineers running multi-agent AI workflows and SREs troubleshooting production runs will get faster, more targeted debugging: find the responsible agent and the earliest actionable step to fix. Research teams and tool builders aiming to add monitoring, agent reputation, or automated post-mortem tooling can use this approach to generate richer supervision and improve localization models. Multi-Agent Scientific Research
Key Figures
Fig 1: Figure 1: Illustrative example of failure propagation in a multi-agent system.
Fig 2: Figure 2: Distribution of step localization error.
Fig 3: Figure 3: Localization performance of different methods with different step tolerance parameters.
Fig 4: Figure 4: Distribution of failure trajectories by token length.
Ready to evaluate your AI agents?
Learn how ReputAgent helps teams build trustworthy AI through systematic evaluation.
Learn MoreYes, But...
Localization depends on the Judge and Evaluator LLM quality; weaker models will reduce reliability and may need more evaluator diversity or human review. The method can misattribute failures that only surface late in the pipeline—agents that detect problems are sometimes mistaken for the root cause. Experiments use two public benchmarks and specific instruction-tuned models, so results may vary for different domains, agent designs, or non-text tool interactions. Context Drift
Methodology & More
AgentLocate treats failure localization as a process you can challenge and refine rather than a single prediction. First, an LLM-based Judge reads the failed interaction trace and proposes an (agent, step) pair where changing that single action would have reversed the failure. Second, multiple independent Evaluators—each prompted in a different style—re-analyze the same trace conditioned on the Judge’s hypothesis, returning alternative predicted locations, rationales, and confidence scores. AgentLocate aggregates those votes with confidence weighting to reach a verified decision. Finally, the system converts the Judge hypothesis plus Evaluator feedback into fine-tuning data and adapts the Judge using parameter-efficient fine-tuning (LoRA), improving future localization. Planning Pattern Emergence-Aware Monitoring Pattern
Avoid common pitfallsLearn what failures to watch for
Credibility Assessment:
Authors have low h-indices (mostly <10), no affiliations listed and it's an arXiv preprint — emerging/limited information but some multi-author effort.