In Brief
ASCon uses step-by-step and agent-level context together to accurately identify the responsible agent, the faulty steps, and the failure mode — boosting attribution accuracy by double-digit percentages in key metrics.
ON THIS PAGE
Key Findings
Modeling the execution trace as two directed graphs (one over steps, one over agents) and letting step and agent representations inform each other yields much clearer failure signals. Separately modeling what led up to a step and what came after it prevents mistaking consequences for root causes. Aggregating an agent’s steps into a behavior-aware agent profile and feeding that back into step representations improves who/when/why attribution across benchmarks. Adding ASCon to existing LLM-based checks also improves out-of-domain performance.
Data Highlights
15.83%+ improvement in agent-level micro-accuracy over strong baselines.
210.63%+ improvement in step-level micro-accuracy compared to prior methods.
314.73%+ improvement in failure-mode macro-F1 on benchmark evaluations.
Implications
Engineers building systems where multiple AI agents coordinate — for example, toolchains that split tasks across models — will get faster, more actionable debugging signals. Site reliability and monitoring teams can use the attributions to prioritize fixes and reduce time spent manually reading long traces. Researchers can use ASCon as a modular way to add structure-aware diagnostics into larger evaluation pipelines.
Explore evaluation patternsSee how to apply these findings
Key Figures
Fig 1: Figure 1: (a) MAS failure attribution are commonly formulated at separate agent, step, and failure-mode levels. (b) These targets share the same trajectory evidence, which ASCon models for unified attribution across all three levels.
Fig 2: Figure 2: The framework of our ASCon model.
Fig 3: Figure 3: Performance comparison between vanilla graphs and LLM-inferred graphs.
Fig 4: Figure 4: Performance comparison on the alert accuracy (%).
Ready to evaluate your AI agents?
Learn how ReputAgent helps teams build trustworthy AI through systematic evaluation.
Learn MoreYes, But...
Results come from two public benchmarks that cover coding, math, and general agentic tasks; performance on very different domains (e.g., real-time robotics) is untested. The approach assumes detailed, timestamped execution logs and explicit agent identities — noisy or incomplete logs will weaken results. [Direction-aware attention and graph processing] add compute and implementation complexity compared with simple prompt-based checks, so integration cost should be weighed against expected debugging savings.
Methodology & More
ASCon treats a failed multi-agent execution like a story with two linked timelines: the timeline of steps (what happened when) and the timeline of agents (who acted when). Each timeline becomes a directed graph where incoming neighbors (what led to a step) and outgoing neighbors (what followed) are handled separately. A direction-aware attention layer keeps preceding and succeeding context distinct so downstream effects aren’t mistaken for root causes. Steps performed by the same agent are aggregated into a behavior-aware agent representation using a masked attention mechanism, then injected back into the step nodes. That reciprocal flow — steps informing agents and agents informing steps — produces richer, contextualized features for attribution. Evaluation on the TracerTraj and Aegis-Bench datasets shows consistent gains: single-model improvements of roughly 6–15% on core attribution metrics, and meaningful out-of-domain boosts when ASCon augments existing language-model-based attribution methods. Practically, ASCon helps pinpoint who is responsible, which steps first showed the error, and the likely failure mode (for example, wrong assumption versus tool misuse). The model is most useful where detailed logs and clear agent roles exist; teams should budget for modest extra compute and engineering work to construct the directed graphs and integrate the attention layers into their monitoring or postmortem pipelines. attribution
Avoid common pitfallsLearn what failures to watch for
Credibility Assessment:
Authors have generally low h-indexes and no strong affiliation signals; arXiv preprint with limited reputation indicators.