Key Takeaway
A new family of game designs can reliably map how many steps of strategic reasoning an AI model uses up to 9–10 steps; across four modern models, their stated chain-of-thought usually matched their actions, but some models lost depth at higher steps.
ON THIS PAGE
What They Found
Games can be designed so each level of iterative thinking produces a unique, observable action; the Ring 11–20 game is one such construct with a 10-step cycle that makes depths 0–9 identifiable. Testing four contemporary models under two information conditions (recursive language about the opponent vs. only sampled actions) shows high internal consistency between what models say they reasoned and what they actually did. Model-specific failures appear at higher depths: one model falls one level short past depth 4, and another shows lapses at depths 9–10. Tree of Thoughts Pattern
Data Highlights
1Ring 11–20 produces 10 distinct iterated responses (period = 10) and is distinguishable across depths 0–9 (injective up to K_max = 9).
2Four LLMs were evaluated (Gemma 4 31B, Qwen 3.6 27B, Claude Sonnet 4.6, DeepSeek v4 Pro) across target reasoning depths 1–10.
3Model breakdowns: Sonnet 4.6 produced actions one level short beyond depth 4; Gemma 4 showed failures at depths 9–10 (loss of capacity at the top depths).
Why It Matters
Engineers building multi-agent systems and researchers measuring agent reliability will get a practical, reliable instrument for estimating how deeply a model reasons about other agents. Technical leaders responsible for agent trust or safety can use these distinguishable games to detect when an agent claims deeper planning than its actions support and to calibrate agent-to-agent evaluation and delegation policies.
Explore evaluation patternsSee how to apply these findings
Key Figures
Fig 2: Figure 2: Recursive condition: Depth limit—linguistic and behavioral signals as a function of target depth. Each panel is one model; solid lines and markers show K B K_{B} (behavioral depth), dashed lines show K L K_{L} (linguistic depth), and the gray dashed diagonal marks where K K equals target k k .
Fig 3: Figure 3: Inductive condition: Depth limit—linguistic and behavioral signals as a function of target depth. Columns are models and rows are the sampling precision λ \lambda ; solid lines and markers show K B K_{B} (behavioral depth), dashed lines show K L K_{L} (linguistic depth), and the gray dashed diagonal marks where K K equals target k k .
Fig 4: Figure 4: Per-step best-response consistency and chain length against target depth k k , over recursive-condition trials, one panel per model. The solid line with markers is per-step consistency on the left axis—the share of adjacent steps in a model’s written run that are exact best responses—with 95% confidence intervals. The heavy dotted line is mean chain length on the right axis, and the grey dashed diagonal is the target length, m = k m=k . Both series break between k = 4 k=4 and k = 5 k=5 , where the form of the opponent description changes: chain length is drawn as two separate segments, and the consistency line runs straight from k = 3 k=3 to k = 5 k=5 with k = 4 k=4 hung off it on fine dotted spurs.
Ready to evaluate your AI agents?
Learn how ReputAgent helps teams build trustworthy AI through systematic evaluation.
Learn MoreConsiderations
The tests only probe up to ten iterative steps, so conclusions don’t extend to deeper reasoning. A single LLM was used to read and score chain-of-thought traces, so judge errors could influence results. Experiments covered four models and a handful of adapted games, so behavior may differ on other architectures or with different anchoring actions. Emergence-Aware Monitoring Pattern
Deep Dive
Level-based models of bounded reasoning label agents by how many steps they iterate when best-responding to others. Many standard games used to infer that depth map different depths to the same action, making depth unidentifiable from behavior alone. To fix that, construct games whose iterated best-response sequence produces unique actions at each depth — a property called level-k distinguishability. Ring 11–20 is a five-player adaptation of the classic money-request game that yields a 10-step, non-repeating iterated sequence (period = 10), making depths 0–9 uniquely observable.
Using Ring 11–20 and three adapted two-player games, four large language models were tested across target depths 1–10 under two conditions: recursive (the opponent’s level described in language) and inductive (only sampled opponent actions). Two signals were tracked: the behavioral depth (where the submitted action sits in the iterated sequence) and the linguistic depth (the depth the model says it expects and responds to). Across almost the full range, models’ linguistic and behavioral signals tightly matched, indicating internal consistency between stated reasoning and actions. Still, model-specific drop-offs appeared: one model (Claude Sonnet 4.6) tended to act one level lower past depth 4, and another (Gemma 4) lost capacity at depths 9–10. These instruments give practitioners a clearer, validated way to measure strategic depth for multi-agent trust, Model Context Protocol (MCP) Pattern, and safety calibration. Capability Attestation Pattern
Avoid common pitfallsLearn what failures to watch for
Credibility Assessment:
Authors from a recognized university (University of Virginia) with moderate h‑indices (10 and 8) but posted as an arXiv preprint and no top‑tier venue — solid but not top-tier.