The Big Picture
Co-evolving separate attacker and defender models produces stronger, pointwise safety: attackers find diverse, multi-turn exploits while defenders learn robust refusal behaviors that generalize beyond static red-team data.
ON THIS PAGE
Key Findings
Warming up an attacker with chain-of-thought examples and then alternating attacker/defender training creates a productive feedback loop: attackers generate novel, compositional jailbreaks and defenders learn pointwise-safe replies. Decoupling attacker and defender (separate models and objectives) avoids the optimization conflicts of shared-parameter self-play and yields more stable training. The resulting defenders keep helpful behavior for benign prompts while improving robustness on both single-turn and multi-turn adversarial tests. The approach aligns with the Red Teaming Pattern.
Data Highlights
1Attack pool and warm-up: 20 distinct rewriting strategies; SFT used SorryBench seeds with 440 vanilla prompts and 8,800 adversarial variants to initialize attacker reasoning.
2Supervision scale: the SFT warm-up included 20,000 benign examples alongside the adversarial subset to build generalization before RL.
3Co-evolution training: reinforcement learning used a balanced set of 15,000 harmful + 15,000 benign prompts (30,000 total) to iteratively train attacker and defender models across Qwen2.5 and Llama3.1 families.
What This Means
Engineers building chatbots and agent systems will get a practical way to stress-test and harden models by running attacker-defender co-training rather than relying on static red teams. Security and ML ops teams can use the attack-pool approach to jump-start automated adversary exploration. Researchers tracking model robustness can adopt the asymmetric game view to avoid shared-parameter conflicts in adversarial training.
Avoid common pitfallsLearn what failures to watch for
Key Figures
![Figure 1: Motivation . Left: Static red-teaming and heuristic methods can easily bypass simple filters but fail at complex multi-turn deception due to limited offensive Chain-of-Thought data. Right: Previous self-play methods (e.g., Self-RedTeam [ liu2025chasing ] ) use a single backbone model for both attack and defense, leading to gradient conflicts due to opposing objectives.](https://arxiv.org/html/2602.01539v2/x1.png)
Fig 1: Figure 1: Motivation . Left: Static red-teaming and heuristic methods can easily bypass simple filters but fail at complex multi-turn deception due to limited offensive Chain-of-Thought data. Right: Previous self-play methods (e.g., Self-RedTeam [ liu2025chasing ] ) use a single backbone model for both attack and defense, leading to gradient conflicts due to opposing objectives.
Fig 2: Figure 2: Overview of MAGIC. The framework operates in two phases: Phase 1 (Initialization) warm-up the attacker via SFT on CoT-enriched data to enable reasoning. Phase 2 (Iterative Co-evolution) employs GRPO to approximate the game equilibrium, alternating between optimizing the defender for robust refusal and the attacker for adaptive jailbreaking strategies.
Fig 3: Figure 3: Heatmap of cross-evaluation results between attackers and defenders during iterative co-evolution.
Fig 4: Figure 4: Evolution of fine-grained attack strategy distributions during RL training. Top : Attacker-base (Qwen2.5-7B-IT as the initial RL attacker). Bottom : Attacker-SFT (no-encode) (SFT-initialized Qwen2.5-7B-IT without encoding-style rewrites).
Ready to evaluate your AI agents?
Learn how ReputAgent helps teams build trustworthy AI through systematic evaluation.
Learn MoreConsiderations
Co-evolution adds significant compute and complexity compared with offline finetuning or inference-time filters, so budget and infrastructure matter. Results depend on the attacker warm-up quality and model capacity — weak attackers limit exploration and overly capable attackers may require stronger defenders. Safety judgments rely on an automated reward model for training and evaluation, so human review is still needed before deployment and extension to multimodal or tool-using agents needs more work [Tool Use Pattern].
Deep Dive
MAGIC frames safety as a two-player, sequential game: an attacker first crafts an adversarial prompt, then a defender replies. Training happens in two phases. Phase 1 warms up the attacker with supervised fine-tuning on a constructed attack pool that contains chain-of-thought rewrites (20 rewriting strategies, built from SorryBench’s 440 seeds and 8,800 adversarial examples) plus benign examples so the attacker can reason about how to rewrite queries. Phase 2 alternates reinforcement learning updates: the defender is trained to produce pointwise best responses given each attacker prompt (generating groups of candidate replies and maximizing safety rewards), then the attacker is trained to find prompts that still succeed against that defender. Group Relative Policy Optimization is used to stabilize learning with grouped candidate sampling. The approach aligns with the Planning Pattern. Because attacker and defender are decoupled (separate models and objectives), MAGIC avoids the gradient conflicts seen when a single model plays both roles. That asymmetric setup approximates a subgame perfect equilibrium notion: defenders learn to be safe for each attacker prompt rather than just on average. Practically, co-evolution discovers novel, multi-turn attack patterns and pushes defenders to learn robust refusal or fallback behaviors while preserving helpfulness on benign tasks. The approach was tested using Qwen2.5 and Llama3.1 families with a 30k-sample RL mix (15k harmful + 15k benign) and a learned safety judge; it improves adversarial robustness but requires careful tuning, computational resources, and human oversight before real-world rollout. A handoff between attacker and defender can be considered via the Handoff Pattern.
Need expert guidance?We can help implement this
Credibility Assessment:
Contains a mix of authors with modest h-indices (e.g., Ying Wen h=10, Ziyu Wan h=7) giving moderate credibility despite arXiv venue.