Key Takeaway
Adding short, taxonomy-guided collaboration instructions to agent prompts raised team success from 60.5% to 70.5%, showing targeted guidance cuts common failure modes and improves outcomes.
ON THIS PAGE
What They Found
AI agents collaborating across four asymmetric tasks succeeded about 60.5% of the time, with big differences by task (34%–80%). Analysis of 1,621 failure instances produced a three-part failure taxonomy—Perception, Adaptation, Coordination—split into six recurring patterns. Appending instructions based on that taxonomy to agent prompts improved average success to 70.5%, and early turns were decisive: conversations still open at turn 12 had only a 37% chance to finish successfully. three-part failure taxonomy and prompt-level interventions helped frame these improvements.
Explore evaluation patternsSee how to apply these findings
Data Highlights
1Overall baseline success across 400 GPT-4o conversations: 60.5%.
2Taxonomy-guided prompt intervention raised average success to 70.5% (a +10 percentage-point improvement).
3If a conversation reaches turn 12 it has only a 37% chance of eventual success, and at turn 20 that drops to 19%.
Why It Matters
Engineers building multi-agent systems and AI teams should care because a small, targeted change to agent prompts meaningfully improves collaboration. Product and reliability leads can use the PAC taxonomy to monitor and diagnose failures (perception, adaptation, coordination) and design guardrails or tests before deployment. PAC taxonomy
Key Figures

Fig 1: Figure 2 . The PARLEY platform while running different tasks. (top-left) Tangram Reference: referential communication, (top-right) Grid Coordination: joint action coordination, (bottom-left) Scene Comparison: perspective-taking, (bottom-right) Map Task: spatial direction-giving. Four screenshots of the PARLEY platform arranged in a 2x2 grid showing its four asymmetric collaborative tasks: (a) Grid Coordination, where agents coordinate to reproduce a target pattern on a shared grid, (b) Map Task, a director-matcher task where one agent guides the other through a spatial layout, (c) Scene Comparison, where agents compare two nearly identical scenes to find differences, and (d) Tangram Reference, where two agents see different symbol grids and must communicate to identify matching symbols.

Fig 3: Figure 4 . Data-driven failure taxonomy: 1,621 failure instances clustered into six patterns. Each point is one failure instance extracted from a conversation, embedded with text-embedding-3-large , and projected via UMAP (cosine distance). Colors indicate cluster assignment via Euclidean Ward linkage hierarchical clustering on the full-dimensional embeddings ( k k =3 top-level categories, each bisected into 2 subcategories). A UMAP scatter plot showing 1,621 failure instances extracted from 400 conversations (both successful and failed), projected onto two dimensions using UMAP with cosine distance. Points are colored by their assigned cluster under a hierarchical taxonomy derived via Euclidean Ward linkage on the full-dimensional embeddings: Adaptation Failures subdivide into Progress Tracking (n=244) and Feedback Integration (n=344), Perception Failures into Visual Interpretation (n=90) and Spatial Reasoning (n=98), and Coordination Failures into Verification (n=563) and Grounding (n=282). Clusters show spatial separation with some overlap, reflecting the continuous nature of failure modes.

Fig 4: Figure 5 . Annotated conversation excerpts illustrating each PAC failure type. Each cell shows the task screenshot embedded within the conversation flow alongside the critical fragment. Highlighted spans mark the specific error. Annotations below each excerpt explain the failure pattern. Tool outcomes are condensed. Columns correspond to the three PAC categories: Perception (P1, P2), Adaptation (A1, A2), and Coordination (C1, C2). A 2$\times$3 grid of annotated conversation excerpts from PARLEY experiments, one per PAC failure type. Each cell shows a task screenshot embedded as an inline message within the conversation fragment illustrating the failure, with an annotation explaining the pattern. Tool outcomes are condensed. Columns correspond to Perception (green), Adaptation (indigo), and Coordination (amber).
Ready to evaluate your AI agents?
Learn how ReputAgent helps teams build trustworthy AI through systematic evaluation.
Learn MoreConsiderations
The study used a single advanced model (GPT-4o) and four controlled puzzle tasks, so measured failure frequencies and gains may not generalize to other models or noisier real-world settings. Visual interpretation failures were concentrated in specific tasks, suggesting some problems stem from perception limits rather than communication alone. The intervention was prompt-level; deeper fixes (model training, tooling, or verification) may be needed for persistent perception or environment-grounding errors. early grounding and explicit verification
Deep Dive
PARLEY is a platform that runs pairs of agents through four asymmetric collaboration puzzles (symbol description, shared-grid coordination, scene-difference spotting, and map navigation). Researchers ran 400 conversations (100 per task) pairing two GPT-4o agents with role-specific prompts and logged turn-by-turn interactions and outcomes. They extracted 1,621 failure instances from conversation traces and used embedding and clustering to produce the PAC taxonomy: Perception (visual and spatial errors), Adaptation (progress tracking and feedback integration), and Coordination (verification and grounding). These six patterns describe why conversations break down, not just what fails. PAC taxonomy and Tool Use Pattern provide practical frames. Key results show wide task variation (Scene Comparison 80% success, Grid Coordination 73%, Map Task 55%, Tangram Reference 34%), and a strong relationship between early grounding and success—short, well-grounded dialogs almost always succeed, while long dialogs usually fail. Agents often behaved differently from humans: they tended to be overly compliant, avoided signaling uncertainty, and repeated failing strategies instead of exploring alternatives. Appending brief, taxonomy-guided collaboration instructions to agent system prompts improved overall success from 60.5% to 70.5%, demonstrating the taxonomy's practical diagnostic and corrective value. For practitioners, the takeaway is concrete: instrument agent conversations for these failure categories, prioritize fixes that improve early grounding and explicit verification, and consider simple prompt-level interventions as a fast, testable step before heavier engineering changes. PAC taxonomy Dynamic Task Routing Pattern Tool Use Pattern ReAct Pattern (Reason + Act) Semantic Capability Matching Pattern
Avoid common pitfallsLearn what failures to watch for
Credibility Assessment:
Authors have low h-indices and no affiliations provided; arXiv preprint with no citation history — fits emerging/limited-info category.