At a Glance
Multilingual requests often lose critical constraints when turned into plans; representing five explicit commitments before planning recovers meaningful accuracy across languages and models.
ON THIS PAGE
What They Found
Five repeatable ways plans lose track of user intent were identified: wrong entity, wrong source, wrong time window, wrong operation, and wrong answer format. Turning a user request into a short, structured checklist short, structured checklist that exposes those five commitments before planning dramatically reduces these errors. The structured representation improves end-to-end task accuracy across three different model families and on two datasets, with larger gains for lower-resource languages.
By the Numbers
1GPT-5-mini: mean exact-match accuracy rose from 24.9% to 30.5% (+5.6 percentage points) across 11 languages when using the structured representation.
2Mistral-Large-3: mean accuracy increased from 19.6% to 25.5% (+5.9 points, a +30.1% relative improvement).
3Per-language boosts reached as high as +10.9 points (Yoruba) and +9.1 points (Kyrgyz) in some model-language combinations.
What This Means
Engineers building multilingual agent pipelines and technical leaders running AI services should care because a small change to how requests are represented can prevent whole workflows from executing the wrong actions. multilingual agent pipelines Evaluation and monitoring teams can use the taxonomy as clearer failure labels for tracking and triage.
Not sure where to start?Get personalized recommendations
Key Figures
Fig 1: Figure 1: We propose a taxonomy for multi-agent planning failures, developed to diagnose multilingual settings, and guide actionable improvements (taxonomy summarized in Fig. 2 ). (a) Distributions of the taxonomy’s planning failures, measured per-language and ordered by Common Crawl language availability (high to low resource). As language-resource availability decreases, taxonomy-covered failures account for an increasing share of unsuccessful executions—particularly operation and entity grounding—while answer formatting failures decrease. (b) Accuracy of OWL on Multilingual GAIA, averaged over the languages. TART (our method) improves all three configurations. Together, the panels connect a systematic multilingual failure pattern to an actionable mitigation that improves a state-of-the-art multi-agent system.
Fig 2: Figure 2: Left: Planner-specific multilingual grounding-failure taxonomy. Right: Entity grounding failure example for an Igbo query. Semantic task representation makes the implicit entity explicit and guides both planning and tool use toward the correct USGS retrieval target.
Fig 3: Figure 3: Taxonomy-guided mitigation via TART. A non-English query is converted by an LLM into a Taxonomy-guided Actionable Task Representation (TART), whose fields correspond to the failure taxonomy, and injected into the planner, coordinator, and worker agents.
Fig 4: Figure 4: Main result. TART consistently improves task-level accuracy on GAIA-MAPS across the three model families. To produce Fig. 1 (b), we averaged per-model across languages.
Ready to evaluate your AI agents?
Learn how ReputAgent helps teams build trustworthy AI through systematic evaluation.
Learn MoreConsiderations
Several evaluated languages were machine-translated, so naturally authored queries might behave differently. The taxonomy was derived from a targeted set of cross-lingual failures, so some English-centric failure modes may be underrepresented. Large-scale labeling relied on an automated judge with partial human validation and repeated-run stability checks only on one model. machine-translated
Methodology & More
Analysis of multilingual agent failures shows that when a user request is converted into an internal plan, key commitments are often lost in five concrete ways: the wrong entity is selected, the specified source gets changed or filtered incorrectly, temporal scope is dropped or altered, the intended operation is misinterpreted, or the answer is formatted incorrectly. Those failures propagate through the multi-agent workflow and cause executions that do not satisfy the original request even if every downstream step runs correctly. A simple, actionable mitigation converts the natural-language request into a short structured representation whose fields map to those five commitments (named TART: Taxonomy-guided Actionable Task Representation). Injecting that representation into the planner and worker agents—without modifying the underlying language models—raises end-to-end exact-match accuracy across datasets and three model families. Gains are consistent across high-, medium-, and low-resource languages, and are largest where language resources are scarcer. The practical implication is that multi-agent systems can improve reliability and auditability by making the planner’s assumptions explicit, producing clearer failure signals for monitoring and simpler fixes for miscoordination. five commitments
Avoid common pitfallsLearn what failures to watch for
Credibility Assessment:
Multiple authors affiliated with Cohere (an established AI company). Overall author h-indices are low, but company backing and mixed experienced contributors justify a strong (4-star) credibility.