Key Takeaway
Routing only matters when models behave differently and treat similar inputs consistently; a carefully chosen small group of models (fewer than 10) captures most useful diversity, and some routing methods that boost accuracy can be brittle to small input changes.
ON THIS PAGE
The Evidence
Two properties determine whether routing is meaningful: behavioral diversity among models, and routing stability (the same query variants should go to the same model). Measuring both with a diversity score (Hierarchic Social Entropy) and a perturbation-based robustness test reveals that most diversity comes from a small curated subset of agents, so huge pools are often redundant. Nearest-neighbor style routers can raise accuracy by exploiting specialist models but lose stability under simple input changes, whereas prompt-based routing stays stable across perturbations even when accuracy differences are small. This aligns with Dynamic Task Routing Pattern which advocates routing decisions based on task properties.
Key Data
1Findings validated on 2 benchmark suites (EmbedLLM and RouterBench).
2A curated subset of fewer than 10 agents recovers most of the society’s behavioral diversity.
3Nearest-neighbor routers improve accuracy with specialist societies but show large drops in robustness under input perturbations; prompted routing remained stable across all perturbation types tested.
What This Means
Engineers designing systems that route work between multiple models should care because a smaller, curated set of specialists can simplify deployment and monitoring. Technical leaders and reliability teams should treat routing stability as a separate signal from accuracy to avoid brittle behavior in production. Researchers evaluating multi-model setups should include diversity and robustness checks, not just task scores and cost. A lightweight Supervisor Pattern can help manage deployments in production.
Test your agentsValidate against real scenarios
Ready to evaluate your AI agents?
Learn how ReputAgent helps teams build trustworthy AI through systematic evaluation.
Learn MoreKeep in Mind
Results come from experiments on the EmbedLLM and RouterBench setups and may vary with other tasks, model families, languages, or perturbation types. The diversity measure and robustness test are diagnostic tools—not prescriptions; how you curate the small agent set depends on your tasks and failure tolerance. Higher task accuracy can mask a lack of meaningful specialization, so relying only on accuracy risks deploying brittle routers. For robustness concerns like drift, see Context Drift.
The Details
Meaningful routing requires two things: models must behave differently enough that assigning different inputs to different models helps, and the router must be stable so that small surface changes to the same query don’t scatter those variants across models. To measure these properties the work adapts a hierarchical diversity score (called Hierarchic Social Entropy) to quantify how behaviorally varied a pool of models is, and introduces a perturbation-based robustness metric that checks whether small input edits lead to consistent actor assignments. Applied to two benchmark collections, the analysis shows strong diminishing returns in diversity as more models are added: a carefully chosen subset of fewer than ten agents recovers most of the behavioral range present in much larger pools. That suggests a practical heuristic—build a small curated ‘core’ of specialists rather than relying on a large, redundant crowd. The study also compares routing methods: nearest-neighbor style routers can leverage specialists to raise accuracy but often fail the robustness test when inputs are slightly changed, while prompt-based routing keeps assignments consistent across perturbations. The upshot for production systems is to evaluate both diversity and assignment stability when designing societies and choosing routers, and to favor curation and robustness checks alongside accuracy. This supports Semantic Capability Matching Pattern to align capabilities with tasks, and cautions against Monoculture Collapse risks in large pools.
Test your agentsValidate against real scenarios
Credibility Assessment:
Contains a very high‑profile author (Mirella Lapata, h‑index 87) at a recognized university (University of Edinburgh). Even though venue is arXiv and citations are 0, strong author reputation justifies top rating.