At a Glance
AI agents often give incompatible probability answers for a claim and its negation, and that mismatch can change later group behavior — so always test consistency separately from overall accuracy.
ON THIS PAGE
What They Found
WHAT THEY FOUND:
Agents answering the same information phrased as a claim or its complement frequently report probabilities that do not sum to one, breaking a basic expectation for binary questions. This mismatch appears in running multi-agent simulations, on real fact-checking claims, and on a controlled benchmark with exact binary alternatives. Forcing consistency by averaging a report with its complemented partner fixes the local mismatch but can help or hurt overall factual performance once those repaired numbers feed back into the system. benchmark
Data Highlights
1Network sweep: 1,728 scenario cells and 124,416 agent updates were evaluated to measure how evidence targeting and update order interact.
2Real-claim study: 48 fact-checked claims produced 3,840 model responses and 1,920 paired complementary requests to measure wording effects.
3Schedule interaction was negligible: primary interaction I = 0.0015 with 95% interval [−0.0093, 0.0132], so update order did not explain the observed wording discrepancies.
What This Means
WHO SHOULD CARE:
Engineers building systems where model-generated probabilities get reused as states (multi-agent orchestration, automated fact-checkers) should care because inconsistent local reports change downstream inputs and behavior. Technical leaders and evaluators should add a complement-consistency check to their evaluation suite before trusting population-level outcomes. multi-agent system
Not sure where to start?Get personalized recommendations
Key Figures

Fig 1: Figure 1: Numerical agent experiments. (a) Agents exchange scalar probability reports on a fixed graph; values and network sizes are schematic. (b) Own state, ordered peer reports, and available evidence enter a fixed checkpoint, which returns one number for the next state. (c) The network study compares three schedules with one-time evidence delivery with one-time evidence delivery. Sequential agents can read already-updated neighbors; displayed identities illustrate order. (d) Targeting contrasts, factual loss, and dispersion are separate outcomes. Complement discrepancy comes from separate paired-request studies: the schedule sweep holds the claim wording fixed. The token example illustrates the controlled benchmark, not the campus claims.

Fig 5: Figure 5: Symmetrization. Left: the same state is described in both orientations and the model answers each request in a separate call, giving q 0 q_{0} and q 1 q_{1} . Middle: q 1 q_{1} is mapped back to the original orientation as 1 − q 1 1-q_{1} and averaged with q 0 q_{0} to give R R . The repaired pair ( R , 1 − R ) (R,1-R) is complementary by construction, and for this fixed pair the average Brier loss falls by δ 2 / 4 \delta^{2}/4 . Right: repaired reports become the next round’s states and are compared with the unrepaired populations on factual loss and call cost. The one-step identity does not determine the outcome of this closed-loop comparison.

Fig 6: Figure 6: Visible-evidence Brier loss after closed-loop feedback on six worlds. Filled circles: loss averaged over the two orientations. Open triangles: the baseline’s original (left) and complementary (right) orientations separately. Symmetrization lowers the averaged loss for all three models but is not always better than the better baseline orientation.
Ready to evaluate your AI agents?
Learn how ReputAgent helps teams build trustworthy AI through systematic evaluation.
Learn MoreYes, But...
CONSIDERATIONS:
The tests used a small set of open models and specific prompt templates, so magnitudes may differ for larger or differently tuned models. Symmetrization (averaging a report with the complement of its pair) enforces local consistency but changes future inputs, and its net effect depends on the model and scenario. Complement inconsistency is distinct from calibration or paraphrase sensitivity; passing one check doesn't guarantee the others. Symmetrization technique
Methodology & More
FULL SUMMARY:
The study defines a simple operational test: if a model sees the same evidence but one request names the claim and the other names the exhaustive opposite, the two reported probabilities should sum to one. Researchers ran three experiments: a network simulation that tracked how evidence targeting and update order affect group outcomes, paired probes that compared real fact-checking claims and their negations, and a controlled benchmark with explicitly exhaustive binary worlds to separate complement issues from paraphrase effects. Across these settings, several small open models produced systematic complement discrepancies — the reported probabilities for complementary descriptions did not add to one.
The authors link local complement consistency to population-level behavior: if every local update were complement-consistent, fully complementing a run (swap claims and flip probabilities) would preserve relative policy comparisons. However, the converse is false, so simulation-level robustness can hide local wording effects. They also evaluate a simple repair: take the two reports for a paired request, map one back into the original orientation, average them, and use the symmetric pair (R, 1−R). That projection always lowers average mismatch for the fixed pair, but when repaired outputs feed back into the simulation the closed-loop factual loss may improve or worsen depending on the model. Practical takeaway: treat generated probabilities as fragile inputs — report both factual performance and complement consistency, and test any repair under closed-loop feedback before adopting it in production. compound AI system Market-Based Coordination Pattern
Avoid common pitfallsLearn what failures to watch for
Credibility Assessment:
No affiliation or author h-index information provided and arXiv-only; lacks identifiable reputation signals.