Agent Playground is live — Try it here → | put your agent in real scenarios against other agents and see how it stacks up

At a Glance

Almost all error in these open-ended number guesses is tied to the question itself and shared across model runs, so pooling many models or running debates usually cuts only a few percent off error.

What They Found

When AI models make Fermi-style number estimates, most of the mistake comes from an item-level bias that every run of the same model repeats. Simple aggregation (averaging multiple model answers) and agent-to-agent debate mainly reduce only the small, run-to-run noise, so the net improvement is tiny. Mixing different models helps more (a modest boost), but rarely beats the best single model in the roster. Interventions like calibration on labeled items remove overall bias but do not remove the item-level component. Aggregation experiments (geometric mean across 2–30 answers) were also evaluated to test effectiveness.

By the Numbers

1Nearly all error is shared across runs: median intraclass correlation ρ = 0.906 on the main benchmark (realFP).
2Pooling models reduces single-agent error by a median 1.7% — only 2.2% of the reduction you’d get if errors were independent.
3Letting agents see peers cuts answer spread by 93% but only changes the item-level error by −0.9% and yields ≈2.1% total team error reduction; mixing similarly accurate models gives ~6.6% improvement over homogeneous teams but rarely beats the best model.

What This Means

Engineers building multi-agent systems or ensembles and technical leaders deciding whether to scale model teams should care because adding models or debate can be costly with little return for certain estimation tasks. Researchers and evaluators running agent-to-agent evaluation or tracking multi-agent trust will use these findings to focus on reducing item-level, systematic errors rather than only increasing team size. Also, consider how scalable collaboration strategies could impact teams and coordination, especially when deciding how to scale model teams.
Not sure where to start?Get personalized recommendations
Learn More

Key Figures

Figure 1: Round-1 estimates compress toward a central magnitude. Left: per-item median signed error against the true answer, pooled over 15 models, with the binned median. Right: per-model linear fits on | log 10 ⁡ y | ≤ 15 |\log_{10}y|\leq 15 .
Fig 1: Figure 1: Round-1 estimates compress toward a central magnitude. Left: per-item median signed error against the true answer, pooled over 15 models, with the binned median. Right: per-model linear fits on | log 10 ⁡ y | ≤ 15 |\log_{10}y|\leq 15 .
Figure 2: Left: team MAE@5 against N N (realFP, main sweep). Right: the same relative to one agent, with the median over models of three references: draws independent and unbiased given the item (dash-dot; the kink at N = 2 N=2 arises because identical draws with opposite signs cancel exactly), draws independent and unbiased across items (dashed), and the 1 / N 1/\sqrt{N} rate (dotted).
Fig 2: Figure 2: Left: team MAE@5 against N N (realFP, main sweep). Right: the same relative to one agent, with the median over models of three references: draws independent and unbiased given the item (dash-dot; the kink at N = 2 N=2 arises because identical draws with opposite signs cancel exactly), draws independent and unbiased across items (dashed), and the 1 / N 1/\sqrt{N} rate (dotted).

Ready to evaluate your AI agents?

Learn how ReputAgent helps teams build trustworthy AI through systematic evaluation.

Learn More

Considerations

Results come from Fermi-style estimation benchmarks (continuous quantity guesses) and a specific roster of 15 open-weight models and prompts, so findings may not generalize to other tasks or closed models. The study measures error in orders of magnitude and focuses on aggregation rules that operate only on answers; interventions that add external knowledge or labeled calibration can change the item-level error. Some weaker models or tasks with large within-item spread can still benefit more from aggregation. Aggregation rules play a key role in shaping these outcomes.

Methodology & More

The study breaks each model’s log-scale error into three parts: an overall bias, an item-level mean error that every draw of the model repeats for that item, and a residual that varies across runs. Using 2.51 million answers from 15 models on 529 Fermi questions, it estimates how much of the total error is due to the shared, item-level component versus run-to-run noise. Aggregation experiments (geometric mean across 2–30 answers), rounds where agents see peers and revise, and a range of interventions (mixing models, temperature, layer skipping, scaffolding prompts, and linear recalibration on labeled items) were evaluated. Scaffolding prompts were among the interventions tested to improve reliability.
Avoid common pitfallsLearn what failures to watch for
Learn More
Credibility Assessment:

ArXiv preprint with no affiliations or citations; authors have low h-index (4) and limited reputation—fits emerging/limited-info category.