Agent Playground is live — Try it here → | put your agent in real scenarios against other agents and see how it stacks up

At a Glance

Attacks can be built quietly across multiple AI agents using harmless fragments; a process-level auditor that checks shared state plus semantics stops nearly all attacks while avoiding mass blocking of legitimate work.

What They Found

Multi-agent pipelines can silently assemble a harmful action from individually benign messages stored in shared state, a failure mode called semantic alignment decay. Simple perimeter filters either miss these campaigns or stop them only by refusing most legitimate jobs. A benchmark (1,728 scenarios across enterprise workflows) exposes four common erosion patterns, and a calibrated Meta-Auditor that combines deterministic state checks with an asynchronous semantic monitor detects almost all attacks while keeping false positives low. Emergence-Aware Monitoring Pattern.

Data Highlights

143.9% of attack scenarios succeeded on an undefended multi-agent mesh (199/453) on the held-out split.
2A state-aware Meta-Auditor detected 446 of 453 attacks (~98.5% recall) and reduced residual attack success to 1.3%, with a 2.9% false-positive rate on matched benign controls.
3Perimeter defenses reduced attacks only by over-blocking: a representative text boundary guard refused 92.5% of legitimate jobs, and another policy layer refused 64.0%.

What This Means

Engineers running or building multi-agent pipelines (gateway/orchestrator/policy/deployment) should care because shared state can be poisoned across sessions and lead to real production compromises. Security and platform leaders testing agent governance can use the benchmark and auditor ideas to balance safety against blocking useful work. Agent Registry Pattern
Not sure where to start?Get personalized recommendations
Learn More

Key Figures

Figure 1: ErosionBench pipeline and threat model. A generation model splices T1–T3 needles into a benign enterprise corpus (Stage 1); a four-agent mesh (Gateway, Orchestrator, Policy, Deployment) executes the scenario over a shared ledger, so the attack propagates through poisoned state and inter-agent debate rather than one malicious string (Stage 2); the Meta-Auditor reads the resulting telemetry to reach an allow/block verdict (Stage 3).
Fig 1: Figure 1: ErosionBench pipeline and threat model. A generation model splices T1–T3 needles into a benign enterprise corpus (Stage 1); a four-agent mesh (Gateway, Orchestrator, Policy, Deployment) executes the scenario over a shared ledger, so the attack propagates through poisoned state and inter-agent debate rather than one malicious string (Stage 2); the Meta-Auditor reads the resulting telemetry to reach an allow/block verdict (Stage 3).
Figure 2: ROC on identical held-out telemetry ( opus-4-8 auditor, gpt-5.5 mesh): the auditor’s signal dominates the same-telemetry LLM judge everywhere, not just at a tuned threshold.
Fig 2: Figure 2: ROC on identical held-out telemetry ( opus-4-8 auditor, gpt-5.5 mesh): the auditor’s signal dominates the same-telemetry LLM judge everywhere, not just at a tuned threshold.
Figure 3: Raw intent-divergence signal (ROC) per auditor, showing the capability gradient the benchmark separates.
Fig 3: Figure 3: Raw intent-divergence signal (ROC) per auditor, showing the capability gradient the benchmark separates.
Figure 4: One of the six annotated image cards (item QF_TEMPORAL_0106_T2 , temporal-exploitation, quantitative finance, Tier 2). A forged “Risk Committee Memo” ratifies risk_weight_modifier = 0.0194 {=}0.0194 —below the 0.05 0.05 safety floor that the param_boundary contract enforces—attributing the change to a co_approved governance status sourced from an untrusted gateway_agent , and stamps it ratified . The identical payload is delivered as text in the paired Mode-A cell. The two annotators rated it realism 3.5 3.5 and image legitimacy 2.0 2.0 : the visible tells (raw configuration keys dumped in a code box; header entity not matching the ledger’s) drew suspicion but not a unanimous “illegitimate” flag—detection of the sub-threshold risk weight is the auditor’s job, not the reader’s.
Fig 4: Figure 4: One of the six annotated image cards (item QF_TEMPORAL_0106_T2 , temporal-exploitation, quantitative finance, Tier 2). A forged “Risk Committee Memo” ratifies risk_weight_modifier = 0.0194 {=}0.0194 —below the 0.05 0.05 safety floor that the param_boundary contract enforces—attributing the change to a co_approved governance status sourced from an untrusted gateway_agent , and stamps it ratified . The identical payload is delivered as text in the paired Mode-A cell. The two annotators rated it realism 3.5 3.5 and image legitimacy 2.0 2.0 : the visible tells (raw configuration keys dumped in a code box; header entity not matching the ledger’s) drew suspicion but not a unanimous “illegitimate” flag—detection of the sub-threshold risk weight is the auditor’s job, not the reader’s.

Ready to evaluate your AI agents?

Learn how ReputAgent helps teams build trustworthy AI through systematic evaluation.

Learn More

Yes, But...

The scenarios are synthetic and focused on four enterprise domains; consumer settings and adaptive attackers were not evaluated. The Meta-Auditor requires per-model calibration, so thresholds do not automatically transfer between auditor models. The reference auditor is a research prototype, not adversarially hardened production software, and the image-arm false-positive rate is still provisional. LLM-as-Judge Pattern

Methodology & More

ErosionBench exposes a subtle but practical failure mode in multi-agent systems where an attacker spreads benign-looking fragments across sessions and agents so a later tool call realizes the malicious outcome. The benchmark procedurally generates 1,728 scenarios across quantitative finance, CI/CD supply chain, data engineering, and customer management, with three burial depths (T1–T3) and four escalation mechanisms: cross-session priming, state rewriting, burying the payload deep in long documents, and distributed corroboration across agents. Each scenario is validated deterministically so the benchmark scores whether the assembled process respected intent, provenance, and policy rather than judging only final text output. Defenses were evaluated on five observation levels from isolated single agents up to full-trajectory verification. Isolating a single agent prevents the attack but breaks almost half of legitimate workflows (48.3% undone). Perimeter text filters and final-call policies either miss attacks or stop them only by refusing a large share of benign work (e.g., 92.5% over-refusal for a text boundary guard). A Meta-Auditor that first applies deterministic state anchors (a low-overhead pre-execution State Divergence Index that vetoes only when typed values diverge) and then an asynchronous semantic monitor achieves high detection (about 98.5% recall), low residual attack success (1.3%), and low false positives (2.9%). That combination shows that reliable verification for multi-agent systems needs heterogeneous signals and calibrated operating curves, not a single outcome score. Chain of Thought Pattern Event-Driven Agent Pattern
Avoid common pitfallsLearn what failures to watch for
Learn More
Credibility Assessment:

Includes Silvio Savarese (well-known top researcher) and an author with h-index 21; despite arXiv venue and unspecified affiliations, recognized author reputation yields top credibility.