At a Glance
Per-step or message-level monitors fail when harmful behavior only emerges after pieces are combined; detection returns only if the monitor reconstructs the assembled representation that exposes the payload.
ON THIS PAGE
What They Found
A local monitor that inspects each message or tool output separately can be entirely blind if each fragment looks benign on its own and contains no leftover clue. When fragments are reassembled into the final payload, structural signals (for example, code-like structure) can reappear and be detected by a detector that inspects the assembled view. Simply seeing more fragments or using standard decoders or an LLM probe rarely recovers the needed representation; only a gate that inspects the correctly decoded assembled representation reliably blocks the attack emergence-aware monitoring pattern.
By the Numbers
1A trajectory-length–only classifier reached AUROC 0.940 on some attacks but collapsed to chance under length-matched variants, showing raw local scores can be misleading.
2A one-class, marker-free monitor achieved mean AUROC 0.874 on assembled outputs (no attack training), proving the signal returns once assembled structure is exposed.
3A codec-blind monitor that saw all fragments blocked 0/50 attacks, while a decoded-view gate (given the correct encoding) blocked 50/50, showing breadth alone did not help—representation recovery did.
What This Means
Engineers building agent teams and tool-using language systems: design monitors that can inspect assembled representations, not just individual messages. Security and product teams running production agent monitoring: evaluate whether your runtime checks can reconstruct or access the representation where harm forms, because high per-step detection rates can be a false comfort. Guardrails pattern
Not sure where to start?Get personalized recommendations
Ready to evaluate your AI agents?
Learn how ReputAgent helps teams build trustworthy AI through systematic evaluation.
Learn MoreKeep in Mind
Experiments focus on code-like payloads and controlled encoding families; fully covert natural-language variants remain an open challenge. Results are measured on a specific set of models and encodings, so performance may vary with other architectures or novel carriers. Practical defenses will need ways to discover or reconstruct exposing representations without relying on a known decoder in hand, which is nontrivial in real systems. Red Teaming Pattern
Methodology & More
Built a controlled testbed that keeps the final malicious payload fixed while varying how much of that payload remains visible inside each fragment. Attack fragments were hidden inside benign carriers across several encoding families, including a no-marker family that makes each fragment read like ordinary text. The evaluation compared per-step monitors, taint-style checks, marker-based searches, and a marker-free one-class detector that inspects the assembled output for structural cues such as code-token density and parseability.
Key findings: when each fragment is locally benign and leaves no usable clue in the monitor’s view, no local detector—however strong—separates attack from benign traffic. The assembled view can restore the signal: a one-class marker-free monitor reached mean AUROC 0.874 on assembled outputs without ever seeing an attack during training. However, simply widening the monitor’s view, using standard decoders, or prompting an LLM to search for encodings mostly failed to recover the exposing representation; a codec-blind full-trace window blocked 0/50 attacks in a tested boundary case, while a decoded-view gate that inspects the correct decoded representation blocked all 50/50. Implication: monitor design and evaluation must treat the observation window and representation as security parameters — detecting compositional harm is about reaching the representation where the harm forms, not just aggregating fragments. Capability Discovery Pattern Guardrails pattern
Avoid common pitfallsLearn what failures to watch for
Credibility Assessment:
No affiliations or h-index information provided and authors are not recognizable from training data; limited signals of reputation suggest emerging / limited credibility.