The Big Picture
Training language models inside short games of conversation teaches them lasting strategies: targeted reward shaping that uses three behavioral signals lets models influence opponents and improve multi-turn outcomes.
ON THIS PAGE
Key Findings
Training models in multi-turn conversational games produces better strategic behavior than single-turn fine-tuning. Three behavioral signals—whether the model understands its opponent, whether its actions are effective given its beliefs, and whether it can change the opponent’s behavior—help diagnose weaknesses and guide reward shaping. Using these shaped rewards while fine-tuning led to consistent gains across three representative games, with one adapted fine-tuning method standing out for reward shaping patterns persuasion tasks.
By the Numbers
1Evaluated on 3 games: Rock–Paper–Scissors, Bertrand competition, and a Size–Price bargaining game.
2Training uses k=3 parallel rollouts when duplicating conversation branches for learning.
3Compared 3 adapted fine-tuning methods (GRPO, DPO, STaR); DPO (direct preference optimization) ranked best for persuasion-style tasks.
Why It Matters
Engineers building AI agents for negotiation, customer support, or multi-agent coordination — use these ideas to teach models to plan over many turns instead of replying one line at a time. Technical leaders deciding how to evaluate or deploy conversational agents can use game-based tests and the behavioral signals as targeted benchmarks of long-term strategic ability. multi-agent coordination
Test your agentsValidate against real scenarios
Key Figures
Fig 1: Figure 1: Example of a game of rock-paper-scissors. All texts have been shortened. The game starts with a setting prompt . The two agents first engage in a conversation, using the communication action. When the conversation is over, they do the game action . That concludes the game. Before each of these actions, they use the private CoT , to decide their actions, this is not shown to the opposite LLM. At the end of the game, one LLM is trained using the reward obtained from this episode.
Fig 2: Figure 2: Illustration of how π ^ t r u e \hat{\pi}_{true} and π ^ b e l i e f \hat{\pi}_{belief} are obtained, in order to use them in ISE , SRP and LO . Note that they are approximations, since they have to be computed without the Private CoT , which would influence the final action.
Fig 3: Figure 3: Illustration of the generation process used in GRPO and DPO. The first three interactions correspond to the root conversation, which is then duplicated into k = 3 k=3 parallel copies. Each copy is completed independently, and the reward obtained from each is used to train the policy on the first response after the copying step.
Fig 4: Figure 4: Analysis of reward shaping in the constrained Rock-Paper-Scissors game. Left: Spider plot with general metrics (rescaled for visualization), comparing an Untrained model, a Base agent trained only on game reward, and agents trained with ISE and LO auxiliary rewards. Middle: Bar chart comparing win/draw/lose rates of all models. Right: Spider plot comparing Base and LO-reward against an agent trained with LO-reward and Naturalness reward . Bottom: Table with the raw values of the plots, bold for the best value and underline for the second best.
Ready to evaluate your AI agents?
Learn how ReputAgent helps teams build trustworthy AI through systematic evaluation.
Learn MoreYes, But...
Results come from controlled game environments, so performance may differ in open-ended real-world conversations. Training used a fixed opponent LLM, which encourages robustness to imperfect counterparts but risks overfitting to that opponent style. The reward-shaping signals require careful design and estimation; they improve learning here but may be sensitive to how beliefs and opponent models are approximated. Guardrails Pattern
Deep Dive
GameTalk trains language models inside short, rule-based conversational games so success can be measured by game outcomes rather than single-turn scores. Each episode gives agents private information and alternates private reasoning (a "chain of thought") with public messages and game actions. Learning optimizes the final episode reward (the game utility) but also leverages three behavioral signals: Internal State Evaluation (does the agent predict the opponent’s strategy?), State-Relative Performance (are its actions effective given its beliefs?), and Leverage Opportunity (does it steer the opponent’s behavior?). These signals both diagnose weaknesses and serve as auxiliary shaped rewards to guide learning.
The framework adapts three post-training methods to the conversational-game setting and trains one agent against a fixed LLM opponent to encourage strategies that work against imperfect partners. Experiments on three games show consistent improvements over untrained baselines; targeted reward shaping using the behavioral signals yields the largest gains, and the direct preference optimization variant performed best in persuasion-style tasks. The work shows a practical path to teach models to use language as a strategic tool — useful for negotiation bots, automated assistants, and multi-agent systems — while highlighting limits from simplified games, opponent selection, and the need for careful signal estimation when moving to messy real-world settings. post-training methods Chain-of-Thought Pattern
Test your agentsValidate against real scenarios
Credibility Assessment:
Includes Mihaela van der Schaar, a top researcher; high author reputation warrants top rating.