The Big Picture
AI models often follow the rules but still make poor strategic choices: they match a chance-aware optimal player only 40–46% of the time, and simple narrative framing can flip decisions by up to 33%.
ON THIS PAGE
The Evidence
A spot-based benchmark called LudoBench uses 480 handcrafted Ludo board situations to probe strategic decision-making under chance and multi-piece coordination. Models reliably identify legal moves at different rates, but legal play does not equal good play — models cluster into two incomplete strategies: "finishers" (complete pieces) and "builders" (develop new pieces). No model balanced both tendencies like the chance-aware game-theory baseline, and persona or narrative prompts often had weak or inconsistent effects on behavior. LLM-as-Judge Pattern.
Data Highlights
1Models agreed with the chance-aware baseline only 40–46% of the time, so over half of moves were strategically suboptimal.
2Illegal-move rates fell into three tiers: under 1% for top models, about 4–10% for mid-tier models, and ~40% for the weakest model.
3Telling a model about prior conflict on the same board changed its decision up to 33% of the time (narrative framing effect).
What This Means
Engineers building AI agents should care because rule-following alone won’t produce robust, balanced strategies in multi-agent settings. Technical leaders and product owners should treat vague persona prompts as unreliable and prefer concrete, testable instructions or game-theory informed policies. Researchers studying agent interaction or trust can use spot-based scenarios to reveal behavioral biases that full-game win rates hide. Consensus-Based Decision Pattern
Not sure where to start?Get personalized recommendations
Key Figures
Fig 1: Figure 1: Annotated Ludo Board. Main track (0–51), colored home paths, safe squares, and base areas. Player start positions are P0 = 0, P1 = 13, P2 = 26, P3 = 39.
Fig 2: Figure 3: Average Invalid Rate and Reliability (1 − - invalid rate) across all non-grudge categories. Three compliance tiers are visually evident.
Fig 3: Figure 5: Archetype Clustering: Builder versus Finisher. Each model is plotted by development tendency (x-axis) versus completion tendency (y-axis). The GT agent (star) exhibits both tendencies. LLMs split into finishers (top-left) and builders (bottom-right).
Fig 4: Figure 7: Tradeoff Preference Hierarchies. Each bar shows a model’s preference (left = capture, right = alternative). The green dashed line marks the GT choice. Bars on the same side as the line indicate correct decisions.
Ready to evaluate your AI agents?
Learn how ReputAgent helps teams build trustworthy AI through systematic evaluation.
Learn MoreConsiderations
Scenarios are handcrafted and cover only a subset of possible Ludo decisions, so results are suggestive rather than exhaustive. Each category uses 40 spots (cost-limited), so the reported rates are point estimates without confidence intervals. Evaluation is spot-based (single decisions), English-only, zero-shot at temperature 0, and may differ under multi-turn play, other languages, or alternative model settings. Reflection Pattern
Methodology & More
LudoBench presents isolated single-decision Ludo board situations (480 spots across 12 decision types) so each choice isolates a strategic tradeoff under dice-driven uncertainty. A chance-aware search agent provides a game-theory reference, and experiments run zero-shot on six contemporary models under five persona prompts plus paired "grudge" narratives that test history sensitivity. Each model had its outputs checked for legality and compared to the oracle move to produce per-category behavioral signals like capture preference, safety choices, and change rates. Key findings show a gap between following rules and playing well: some models nearly never make illegal moves yet still adopt opposite and incomplete strategies. Models split into two archetypes — one that prioritizes finishing pieces and one that prioritizes developing new pieces — while the chance-aware baseline balanced both. Agreement with the baseline sits at only 40–46%, illegal-move rates vary widely across models, and simple narrative framing can alter choices by up to a third. Practical implications include treating persona instructions as unreliable for fine-grained strategic control, using scenario-level tests to catch behavioral blind spots, and extending evaluation to full multi-turn play to measure cumulative effects. Market-Based Coordination Pattern Orchestrator-Worker Pattern
Avoid common pitfallsLearn what failures to watch for
Credibility Assessment:
ArXiv preprint, no affiliations provided, and authors have very low h-index (1) or unspecified — minimal reputation signals, likely unknown/solo-level credibility.