At a Glance
Current large language model agents struggle with basic navigation and are effectively at chance when trying to detect deception in multi-agent settings; fixing navigation exposes the deeper social-reasoning failure.
ON THIS PAGE
What They Found
Agents delivered by today's large language models fail on two fronts in an embodied, multi-agent game: poor spatial planning and near-random social deduction. A symbolic navigation helper (Planning Oracle) fixes many movement and task failures but doesn't improve the ability to spot impostors. Even the biggest models rely on shallow heuristics instead of accumulating behavioral evidence, so deception detection remains unsolved.
Not sure where to start?Get personalized recommendations
By the Numbers
1Average impostor detection accuracy across evaluated models: 29.9% (random baseline ≈ 33%).
2Best model (GPT-OSS-120B) completed only 50% of tasks without navigation help; with the Planning Oracle it reached 100% task completion.
3Without assistance path-efficiency scores rarely exceed 0.2; with assistance most models still stay below 0.5, and impostors win >80% of matches in many setups.
What This Means
Engineers building multi-agent systems and agent orchestration platforms need to know that navigation issues and social reasoning failures are separate problems—fixing one won’t fix the other. Technical leaders evaluating agent reliability or pre-production testing should use embodied, adversarial benchmarks like SocialGrid to reveal hidden failure modes before deployment. (embodied, adversarial benchmarks)
Key Figures
Fig 1: Figure 1 : SocialGrid Overview. Inspired by Among Us , SocialGrid is a controllable, embodied benchmark evaluating LLM agents in multi-agent, multi-objective environments. (Left) User Input: Enables systematic control of environmental complexity (e.g., map area, room count) and agent configuration. (Center) Environment: Agents operate under physical constraints; an optional Planning Oracle isolates social reasoning from navigation. (Right) Evaluation & Feedback: Provides multi-dimensional metrics across spatial, task, and social axes, with automated failure analysis and league-based Elo rankings. See Appendix A for visualization screenshots.
Fig 2: Figure 2 : LLM agents struggle with spatial navigation in embodied settings. Comparison of crewmate performance on SocialGrid under low (no assistance) and high (with planning oracle) conditions. 7 crewmates per episode; 20 episodes per model; error bars show SD. Green values indicate absolute improvement from low to high. Left: Task performance (completion rate). Middle: Planning success rate (tasks reached). Right: Planning efficiency (path optimality). Even the strongest model (GPT-OSS-120B) completes only 50% of tasks without assistance; with assistance, performance improves substantially but path efficiency remains below 0.5 for most models.
Fig 3: Figure 3 : Detection accuracy reveals below-random performance across all models. Heatmap showing crewmate detection accuracy across 36 matchups (30 cross-model league + 6 self-play diagonal). All models perform near or below the random baseline (33%, shown in colorbar), averaging 29.9% detection accuracy. The consistent near-chance performance indicates that impostor detection remains challenging regardless of model pairing.
Fig 4: Figure 4 : Trust calibration hovers near random baseline. Trust metrics for crewmate models facing GPT-OSS-120B impostor. Left : Brier Score measures how well trust predictions match ground truth (lower is better); most models hover near the random baseline (0.33, dashed). Right : Volatility measures how erratically trust changes between turns (lower is better); values around 0.33 indicate unstable assessments.
Ready to evaluate your AI agents?
Learn how ReputAgent helps teams build trustworthy AI through systematic evaluation.
Learn MoreConsiderations
Results come from a discrete gridworld with text-rendered state descriptions and simplified game rules (no spoken discussion phase), so outcomes may change in richer visual or continuous environments. Game balance choices (e.g., impostor advantages, kill cooldowns) can amplify certain failure modes, so interpreting low detection rates requires care. Prompting strategy, memory augmentation, or vision-enabled models were not exhaustively explored and might change performance profiles. (Prompting strategy) and (vision-enabled models)
Methodology & More
SocialGrid is an embodied, configurable benchmark that combines movement, task completion, and adversarial social reasoning in a single grid-based environment inspired by social deduction games. Agents perceive local views, take physical actions, and must both complete tasks and identify hidden impostors who sabotage the group. The benchmark runs league-style matches and reports separate metrics for spatial planning, task success, and social detection, plus automated failure analysis to pinpoint behaviors like stalling, oscillation, or door-spamming. Across eight public large language models (14B–120B), navigation and task completion were weak without a symbolic navigation assistant; adding a Planning Oracle fixed many spatial failures but left deception detection at near-random levels (≈29.9% accuracy). Analysis shows agents use brittle heuristics—e.g., reacting to single events rather than aggregating behavior over time—which explains why social reasoning doesn't improve even when navigation is solved. The takeaway for practitioners is to evaluate embodied agents with tests that separate spatial competence from social inference; tool-based fixes can patch one axis but reveal deeper deficits in theory-of-mind and trust calibration that require new model capabilities or architectures. (Planning Oracle)
Avoid common pitfallsLearn what failures to watch for
Credibility Assessment:
All authors have low h-indices and no listed institutional affiliations; venue is arXiv with zero citations. Signals point to emerging/limited-info rather than established lab/researcher.