In Brief
Targeting prompt edits to the specific agent roles and interaction rounds that cause failures improves multi-agent reasoning accuracy and cuts wasted search effort compared with global, black-box tuning.
ON THIS PAGE
Key Findings
Adding a simple per-round summary step and forcing each role to share the same prompt across rounds makes it possible to tell which agent or which stage caused a wrong answer. Using lightweight critics (other language models) to score those intermediate states lets you focus edits on the worst roles or the weakest rounds instead of changing everything. Alternating updates between role prompts and the per-round aggregator prompt (an algorithm like block coordinate descent) gives more stable, faster improvements than joint or blind search. Evaluation-Driven Development (EDDOps)
Key Data
1100 optimization examples used per dataset for prompt search (test sets were held out and never used during optimization).
2Evaluation ran on 4 benchmark suites: AQuA, MedMCQA, GPQA, and MMLU across 3 model families (Qwen2.5-7B, LLaMA3-8B, Gemma-7B).
3Structural updates targeted the two worst-performing roles per run (deterministically selected), reducing the number of edited prompts compared to global search.
What This Means
Engineers building systems where multiple AI agents collaborate (planners, debaters, synthesizers) — you can diagnose which role or which round causes failures and spend your tuning budget on those weak links. Technical product leads and reliability teams benefit because focused edits reduce wasted evaluation cost and make agent behavior easier to audit and justify for trust and governance. Human-in-the-Loop
Explore evaluation patternsSee how to apply these findings
Key Figures
Fig 1: Figure 1: Overview of the credit-guided prompt optimization pipeline. Top: a multi-agent, multi-round reasoning loop (planner/solver/critic) produces per-round messages that are aggregated into a shared system state S r S_{r} ; an aggregation module feeds back to the next round. From the completed trajectory, we compute temporal credit across rounds (identifying critical rounds) and structural/spatial credit across agents (identifying effective or limiting roles). These credits then drive inference-time prompt updates, selectively refining the lowest-credit rounds/roles while keeping strong components fixed. Bottom: an example travel-itinerary task illustrates per-round agent outputs, the evolving shared state ( S 1 , S 2 , … S_{1},S_{2},\ldots ), temporal credit weights, and a before/after prompt update that specializes guidance to the weak round/role.
Fig 3: Figure 3: LLaMA3-8B Debate results on MedMCQA and MMLU: the combined temporal+structural credit-guided optimization yields the strongest gains, with structural-only and temporal-only providing smaller improvements over the baseline.
Fig 4: Figure 4: Accuracy convergence over optimization iterations; ours converges faster and to higher accuracy than DSPy MIPRO and the baseline.
Ready to evaluate your AI agents?
Learn how ReputAgent helps teams build trustworthy AI through systematic evaluation.
Learn MoreYes, But...
Requires completed trajectories and an external evaluator to score intermediate states, so it doesn’t apply when you lack reliable ground truth or evaluative signals. Attribution becomes heavier as the number of rounds grows, making long-horizon interactions more expensive to analyze. The current setup assumes fixed roles and a central aggregator; dynamic role assignment or decentralized aggregation will need extra work to apply the same method reliably. State Inconsistency
Methodology & More
Make multi-agent reasoning systems diagnosable by forcing two structural choices: (1) insert a per-round aggregation step that compresses all agent messages into a single shared state, and (2) share the same role prompt across rounds so each role’s behavior can be measured repeatedly. Those choices create a clear “bottleneck” and recurring measurements that let you separate timing errors (which round failed) from role errors (which agent failed). Lightweight critics based on language models score intermediate outputs and produce per-round and per-role credit signals that point to weak components. Orchestrator-Worker Pattern Consensus-Based Decision Pattern
Optimize prompts by alternating targeted edits: first refine role prompts while holding aggregation logic fixed, then refine the aggregator prompt while holding role behavior fixed — an alternating-update routine similar to block coordinate descent. In practice the workflow uses 100 held-out optimization examples per dataset, selects the two worst roles for structural edits, and only updates an aggregator prompt when its round-level credit falls below a threshold. Across multiple benchmarks and three model families, this targeted approach converges faster and yields more reliable gains than black-box global search, while also giving interpretable failure diagnoses that help with agent trust and monitoring.
Avoid common pitfallsLearn what failures to watch for
Credibility Assessment:
Authors have low h-indices, no notable affiliations or top publication venue — fits emerging/limited-info category.