Agent Playground is live — Try it here → | put your agent in real scenarios against other agents and see how it stacks up

At a Glance

Keeping facts just because they’re rare or just because they’re on-topic can still make an AI team choose the wrong answer; what matters is the balance of retained evidence and how the reader weights who said it.

What They Found

Selecting memory by rarity (keep facts stated by few agents) or by relevance (keep on-topic facts) each fails in predictable cases because they don’t track whether kept facts collectively favor the correct option. Models also overweight facts attributed to many agents, so showing how many agents stated a fact creates a social-proof bias. In real discussions much useful evidence is lost before any memory compression, so no single memory strategy dominates across settings. memory strategy can help frame how compressions should maintain balanced evidence.
Not sure where to start?Get personalized recommendations
Learn More

By the Numbers

1Rarest-first selection reached 88.2% accuracy on Qwen and 87.7% on Llama at the tested budget, beating an evidence-summary baseline by +29.2 and +26.7 percentage points respectively.
2A memory consisting only of private facts scored 91.3% (Qwen) and 90.3% (Llama), while a memory of only shared facts scored 5.6% and 4.6% — both worse than leaving memory empty (35.4% and 36.4%).
3Removing or randomizing the 'who said it' attribution tags raised accuracy in the critical hidden-profile cell by 13 to 24 percentage points, revealing a strong social-proof effect.

What This Means

Engineers building multi-agent systems and teams responsible for agent governance should care because memory-compression rules can flip correct group outcomes into incorrect ones. memory-compression rules Evaluation and monitoring teams should test how evidence ownership and attribution affect downstream decisions and avoid using raw agreement counts as a credibility signal.

Key Figures

Figure 2: Pipeline. Facts from template M01 (Table 4 ) are distributed among the agents; HiddenBench tasks have three or four agents and CrossProfile tasks have four. After the discussion, a memory policy compresses it under a token budget B B , either by summarizing the transcript (B3) or by selecting whole claims from a claim inventory (R1, R1-rel, B4, Random). A separate call with the same model weights answers from the task and the memory alone.
Fig 2: Figure 2: Pipeline. Facts from template M01 (Table 4 ) are distributed among the agents; HiddenBench tasks have three or four agents and CrossProfile tasks have four. After the discussion, a memory policy compresses it under a token budget B B , either by summarizing the transcript (B3) or by selecting whole claims from a claim inventory (R1, R1-rel, B4, Random). A separate call with the same model weights answers from the task and the memory alone.
Figure 8: R1 accuracy at r = .6 r=.6 by support and decoy ownership, separately for each model and irrelevant-fact assignment. Every cell averages the same 60 templates and three seeds; intervals are in Tables 7 and 9 .
Fig 8: Figure 8: R1 accuracy at r = .6 r=.6 by support and decoy ownership, separately for each model and irrelevant-fact assignment. Every cell averages the same 60 templates and three seeds; intervals are in Tables 7 and 9 .

Ready to evaluate your AI agents?

Learn how ReputAgent helps teams build trustworthy AI through systematic evaluation.

Learn More

Considerations

Experiments use scripted benchmarks (CrossProfile and HiddenBench) and a fixed small group size, so results may vary with open-ended real-world conversations or larger teams. Two open models and two hosted models were tested; model-specific differences could change magnitudes. In natural discussions much evidence never reaches memory, so interventions may need to act earlier in communication, not only at compression time. evidence

Methodology & More

CrossProfile is a diagnostic benchmark that fixes the content of facts while crossing whether each fact is shared by all agents or private to one, and whether facts support the correct answer, act as decoys, or are irrelevant. Controlled and natural discussions were run, then memory policies compressed those discussions under a token budget. Policies included rarest-first (keep facts stated by few agents), relevance-first (keep on-topic facts), a full transcript control, random selection, and an oracle relevance control. A separate decision call with the same model weights answered based only on the memory and the task prompt. Rarest-first can look excellent when private facts happen to support the correct answer: it hit ~88% accuracy on two models in a controlled hidden-profile setting and outperformed summary baselines by ~26–29 points. But when ownership is crossed so that rare facts favor decoys, rarity hurts. Relevance-first reintroduces decoy facts because decoys are, by design, on-topic. The downstream decision model also adds bias: it treats facts attributed to many agents as more credible, and removing or randomizing those attribution tags improved accuracy by 13–24 points. In natural discussions the main bottleneck is often upstream (claims never stated or lost before memory acts), so no single memory policy consistently wins. The takeaway for builders is to design memory and presentation so they preserve the balance of evidence between options and to avoid presenting speaker counts as independent support. Rarest-first and memory policies
Avoid common pitfallsLearn what failures to watch for
Learn More
Credibility Assessment:

Multiple authors but no affiliations or citation/h-index signals; arXiv preprint only — minimal credibility indicators.