Key Takeaway
Training each agent to output a calibrated probability alongside its answer gives downstream agents a reliable signal to trust, recheck, or allocate more work, improving overall accuracy and enabling selective answering.
ON THIS PAGE
Key Findings
Agents can learn to express numeric, well-calibrated confidence together with their answers, and downstream agents can be trained to use that signal when deciding to accept or recheck upstream reports. A training pipeline splits behavior into a task segment and a confidence segment, applies separate learning signals, and reads a probability from two tokens so confidence receives direct supervision. Across three collaboration styles (handoff, fusion, review), calibrated confidence improved task accuracy and calibration quality, influenced downstream choices, and produced final confidence estimates useful for withholding uncertain answers or directing targeted rechecking.
Explore evaluation patternsSee how to apply these findings
Data Highlights
1Evaluated on 3 collaboration architectures: sequential handoff, report fusion, and answer review, with consistent improvements across all three.
2Confidence is reported as an explicit probability in [0,1] (read out from two designated tokens) and trained against binary correctness labels, enabling direct probability supervision.
3Models trained on a 2-hop (two-step) collaboration setup transferred to deeper web-retrieval chains and remained effective at smaller model scales.
Why It Matters
Engineers building multi-agent systems: use calibrated confidence to decide when downstream agents should accept or verify upstream work. Technical leaders and product managers: apply confidence-based controls to reduce delivered errors or to route extra computation where it matters. Researchers: this gives a practical recipe to connect per-role calibration with system-level reliability and selective answering.
Key Figures

Fig 1: Figure 1. Confidence at the agent handoff. Without a confidence signal, the downstream agent cannot distinguish a reliable report from an unreliable one. MASCAL attaches calibrated confidence to each report and to the final answer, guiding downstream decisions and enabling reliability assessment of the system output.

Fig 2: Figure 2. The MASCAL training pipeline. (a) Each rollout is split into a task segment and a confidence segment, trained with separate policy gradients and a BCE loss on the readout c c . (b) Masking the upstream confidence reveals which downstream tokens depend on it; those positions are up-weighted during task learning.

Fig 3: Figure 3. Collaboration architectures in our experiments.
Ready to evaluate your AI agents?
Learn how ReputAgent helps teams build trustworthy AI through systematic evaluation.
Learn MoreConsiderations
Experiments use fixed role assignments and binary correctness feedback, so results may not directly extend to heterogeneous agent teams or open-ended outputs where correctness isn't binary. Calibration depends on dedicated supervision and downstream training — simply exposing a confidence number without calibration or downstream learning is unlikely to help. Further work is needed to apply this to multi-round interaction, dynamic role assignment, or systems mixing different model families. non-binary correctness scenarios
Full Analysis
Agents that collaborate over web content become more reliable when each one reports how likely its answer is to be correct. The method trains each agent to produce two outputs: the normal task trajectory (reasoning, tool calls, answer) and a short ‘‘confidence’’ analysis. The numerical probability is read from the model logits for two designated tokens and trained directly against binary correctness labels, while the task and confidence text receive separate policy-gradient signals. This joint training objective produces probabilities whose numerical values match empirical correctness rates (calibration).
To make downstream agents actually use that signal, the downstream role is trained with confidence-aware weighting: masking identifies which tokens in the downstream generation change when upstream confidence is varied, and training upweights the positions that depend most on the upstream probability. Across three collaboration modes (handoff, fusion, review) and tasks involving web retrieval and multi-step reasoning, this setup improved final-answer accuracy and produced better-calibrated upstream and downstream confidence estimates. The learned final confidence can be used to withhold uncertain answers or trigger targeted rechecks, and models trained on two-hop chains generalized to longer retrieval chains and to smaller model sizes. Remaining gaps include heterogeneous role setups and non-binary correctness scenarios, which will need broader calibration objectives and end-to-end training across roles. two-hop chains
Avoid common pitfallsLearn what failures to watch for
Credibility Assessment:
Many authors but low h-indexes (≤5), no prominent affiliations or top venue and no citations — falls into emerging/limited credibility.