Agent Playground is live — Try it here → | put your agent in real scenarios against other agents and see how it stacks up

The Big Picture

Agent collaboration patterns change when work becomes ready to run, and that reshapes batching, memory use, and task completion time even when total calls and tokens stay the same; MAServeBench makes those effects measurable and comparable across serving backends and hardware.

The Evidence

Collaboration rules (who waits for whom, when artifacts are shared, and how many parallel branches run) alter both the concrete calls generated and the moments those calls become eligible to run. A composable stage system produces realistic multi-agent workloads, while hierarchical Semantic Traces record every realized operation and dependency so you can replay the same work under different release rules. Controlled experiments then show how those exposure differences change batching opportunities, memory pressure for key-value context storage, and end-to-end task latency and throughput on GPUs and NPUs. Planning Pattern Dynamic Task Routing Pattern.
Not sure where to start?Get personalized recommendations
Learn More

Data Highlights

1210 native workload realizations created from 30 task bindings to cover diverse collaboration patterns and structure.
2Functional parity checks show task-quality differences of only 1.2–3.8 percentage points across matched native workflows, with at most 0.019 difference on the RACE score, validating replay fidelity.
3Serving and stress tests run at 32 / 64 / 128 concurrent users to reveal how collaboration shape and release rules affect batching, memory reuse, and completion-time SLOs.

What This Means

Engineers building multi-agent systems and serving stacks should use this to evaluate how agent coordination patterns change real serving demand and hardware costs. Technical leaders and SREs can use the benchmark to compare schedulers, caching, and placement decisions under realistic multi-agent traces before deploying to production. Agent Registry Pattern

Ready to evaluate your AI agents?

Learn how ReputAgent helps teams build trustworthy AI through systematic evaluation.

Learn More

Yes, But...

Validation focuses on a representative but finite set of workflows, 30 task bindings, and specific models and serving stacks, so rare production patterns could behave differently. Hardware sensitivity tests used particular GPU and NPU setups and vLLM-based serving; different backends or model architectures may shift absolute costs. Fixed-work replay preserves realized operations and dependencies, but semantic changes to inputs or partial artifact selection require new native realizations rather than replay. Dynamic Task Routing Pattern

Methodology & More

MAServeBench separates what agents are supposed to do (collaboration rules) from what actually runs (realized calls and dependencies) and builds a benchmark around that distinction. Workflows are composed from reusable stages that capture agent roles, handoffs, parallel branches, and refinement rounds; a task binding supplies inputs, tool access, and evaluation criteria independently of those stages. Running a workflow natively produces a concrete trace of model calls, token sizes, and typed dependencies. Hierarchical Semantic Traces persist those realizations so you can replay the exact same work while changing only when operations are legally allowed to run or how the serving backend schedules them. Sub-Agent Delegation Pattern. The benchmark was validated across many structures (210 native realizations from 30 bindings) and compared to multiple agent frameworks to check quality parity (differences of 1.2–3.8 percentage points and ≤0.019 on RACE). Controlled experiments vary release constraints, concurrency (32/64/128 users), and hardware (GPU/NPU) to show that collaboration semantics change batching opportunities, locality for weight reuse, and key-value memory pressure — all of which affect latency and goodput. Use MAServeBench to compare schedulers, caching policies, and placement strategies under realistic multi-agent patterns, and to evaluate agent-to-agent behaviors and trust signals before production rollouts. Planning Pattern
Avoid common pitfallsLearn what failures to watch for
Learn More
Credibility Assessment:

Authors have low h-indexes and no notable affiliations or top-tier venue (arXiv, 0 citations), indicating limited signals of credibility.