The Big Picture
AI agents can automatically tune game rules so two very different factions reach near-equal win rates, turning slow, manual balancing into an interpretable, repeatable search process.
ON THIS PAGE
The Evidence
LLM-driven agents can read a natural-language rulebook and play a turn-based asymmetric game without extra training, producing valid play traces for evaluation. Combining those agent playtests with Bayesian search and an adaptive sampling strategy finds rule settings that drive win rates toward parity (around 50% for each side). The optimized settings also reveal which mechanics matter most—health, resource gathering efficiency, and production speed—giving designers actionable insights rather than opaque tweaks. LLM-as-Judge Pattern.
Not sure where to start?Get personalized recommendations
Data Highlights
1Achieved balanced outcomes within 50% ± 5% win rates for the two factions across validation runs.
2In some runs the win-rate gap was reduced to 0% from deliberately imbalanced starting rules.
3A full optimization run using 8-billion-parameter models for both sides (100 iterations) took ~40 hours on 8 × NVIDIA A100 80GB GPUs.
What This Means
Game designers and production teams who spend weeks on manual tuning can use this to speed iterations, surface interpretable levers, and prioritize human playtests for the most important changes. Researchers and engineers building multi-agent evaluation pipelines can use this as a blueprint for agent-to-agent evaluation and automated rule search in simulations or policy design. Human-in-the-Loop Pattern.
Key Figures
Fig 1: Figure 1: Overview of RuleSmith. Multi-agent LLMs perform zero-shot self-play using solely the rule book under parameterized rule sets to automatically optimize asymmetric strategy games and other rule-driven systems. This figure is generated by Nano Banana Pro.
Fig 2: Figure 2: Overview of the RuleSmith method. We represent an asymmetric, turn-based strategy game (CivMini) as a parameterized rule space θ ∈ Θ \theta\in\Theta , including economy, combat, production, scoring, and game-length parameters. Given a candidate rule configuration θ t \theta_{t} , two role-specific LLM agents ( Empire and Nomads ) play N t N_{t} self-play games in the CivMini environment, producing a balance loss ℒ ( θ ) = | w E − 0.5 | + | w N − 0.5 | + 0.5 ⋅ w D \mathcal{L}(\theta)=|w_{E}-0.5|+|w_{N}-0.5|+0.5\cdot w_{D} , where w E w_{E} , w N w_{N} , w D w_{D} are Empire win rate, Nomads win rate, and draw rate. A Bayesian optimizer maintains a surrogate model g ( θ ) g(\theta) over a continuous relaxation of the rule space and selects new candidates θ ~ t + 1 \tilde{\theta}_{t+1} by maximizing an acquisition function. The number of games N t N_{t} is adaptively determined based on the Expected Improvement: promising candidates receive more games for accurate evaluation. Each continuous proposal is mapped to a valid, discrete ruleset via a deterministic discretization operator D ( ⋅ ) D(\cdot) before evaluation. Cartoons in this figure are generated using ChatGPT-5.2.
Fig 3: Figure 3: Visualization of a CivMini Game. Here we visualize a rollout of an optimized balanced game in which Nomads conquered the Empire’s city and won after 12 turns. Nomads tried to attack Empire’s city from the bottom-right and Empire sent their soldier to defend from top-left. Two Nomads cavalries were stopped, and one was killed, but one managed to escape and arrive at the Empire’s city. Then, after 2 more turns, the Empire city was destroyed, and Nomads won. LLMs for both Nomads and Empire are InternVL3.5-8B.
Fig 4: Figure 4: A rapid Nomad conquest victory achieved in just 6 turns. Nomad Cavalry Unit 0 executed consecutive strikes on the Empire’s city from turns 3 to 6, causing 4 HP damage per turn. This aggression resulted in the destruction of the Empire’s city and an rapid win. (Final score: Empire 15.8 | Nomads 27.6)
Ready to evaluate your AI agents?
Learn how ReputAgent helps teams build trustworthy AI through systematic evaluation.
Learn MoreConsiderations
Results come from a simplified, grid-based prototype game (CivMini), so behavior may change in larger, more complex commercial titles. LLM self-play does not perfectly match human playstyles—some discovered balances may not transfer to human testers without additional validation. The method is compute-heavy (multi-GPU runs) and relies on model capacity matching between training and evaluation for best transferability. State Inconsistency.
Methodology & More
RuleSmith treats a game not as fixed code but as a set of tunable rules (hit points, damage, production cost, scoring weights). Two large language models act as the players: each reads the same natural-language rulebook and generates actions to play full games without extra policy training. For any candidate ruleset the system runs multiple self-play games to estimate win rates and a balance loss that penalizes one-sided results and draws. To search the high-dimensional rule space efficiently, RuleSmith uses Bayesian optimization over a continuous relaxation of the parameters and projects proposals back to valid discrete rule values. An acquisition-driven adaptive sampling step allocates more playtests to promising candidates so evaluation budget focuses where it matters. In experiments on the CivMini prototype, the pipeline consistently found rule configurations that equalized win rates (about 50% ±5%), sometimes eliminating initial imbalances entirely. The discovered changes were human-interpretable—adjusting health scaling, resource flow, and production tempo were the main levers—and the approach suggests broader use for policy or economic simulations where rules drive multi-actor dynamics. Practical caveats include compute cost, sensitivity to the specific language model used, and the need to validate against human behavior for production deployment. Handoff Pattern Agent Service Mesh Pattern.
Avoid common pitfallsLearn what failures to watch for
Credibility Assessment:
Authors mainly have low h-indices (around 2–5), no affiliations, and arXiv-only venue—signals limited established credibility.