The Big Picture
Vision-and-language agents often state facts they never observed; a replayable, statement-level audit is needed because win rates alone hide these groundedness failures.
ON THIS PAGE
The Evidence
QUACK is an open environment that replays every agent action and checks each spoken claim against the true game log. Auditing three strong vision-language agents across many games shows they frequently make spatial claims and accusations that cannot be grounded in what they actually perceived. A statement-level verification pipeline turns vague concerns about hallucination and deception into measurable failure modes that standard game outcomes miss.
Data Highlights
13 frontier vision-and-language models were evaluated across 270 games (homogeneous and cross-model adversarial play).
2Claim-extraction precision validated at 199/200 sampled claims (99.5% correct verdicts), confirming audit reliability.
3Claim-extraction recall validated at 220/223 sampled claims (98.7% coverage); experiments used a single 10×10-room map with 6 players, 1 impostor, and 5 tasks per crew member.
What This Means
Engineers building multimodal agents and evaluators of agent networks should care because groundedness affects whether an agent’s statements can be trusted in interactive settings. Technical leaders and researchers can use QUACK as a diagnostic to surface why two agents with similar win rates might differ sharply in honesty and traceability.
Not sure where to start?Get personalized recommendations
Key Figures
Fig 1: Figure 1: Left: an omniscient view of the game state, from which each agent’s global map I global I^{\text{global}} is rendered (room layout and corridor travel costs, with no other players shown). Top right: the local view I local I^{\text{local}} , rendering only what each agent currently sees. Bottom right: the aligned structured summary τ i t \tau_{i}^{t} . This figure conveys the same semantics as the actual rendered observations the agents receive, but is drawn more cleanly for presentation.
Ready to evaluate your AI agents?
Learn how ReputAgent helps teams build trustworthy AI through systematic evaluation.
Learn MoreYes, But...
Claim extraction is slightly conservative and can miss some utterances, so scored rates may undercount total claims but verdicts are reliable. The study does not isolate the contribution of visual input (no text-only ablation), so failures are attributed to long-horizon, partially observed interaction rather than vision specifically. Experiments use a single map and a one-impostor setting, so absolute numbers may change with different maps, more players, or additional roles.
Methodology & More
QUACK is a replayable, multimodal social-deduction environment that pairs partial observability with an engine-level event log so every agent action and perception can be reconstructed. The evaluation is three-tiered: standard game outcomes (who won), behavioral trajectories (where agents moved and what they did), and a statement-level verification pipeline that parses every discussion utterance into structured claims and checks each claim against the reconstructed ground truth. The pipeline operationalizes four concrete failure modes—spatial hallucination (saying you saw something you didn’t), unsupported accusation (accusing without evidence), deception collapse (lies that are trivially refuted), and language-action inconsistency (statements that contradict recorded actions). spatial hallucination Using the framework to audit three current vision-and-language agents over 270 games, the study finds that strong agents still systematically produce ungrounded statements: many spatial claims and most accusations lack verifiable evidence. Human validation shows the extractor and verifier are highly reliable (≈99.5% precision, ≈98.7% recall). The result is practical: win-rate metrics alone can’t reveal whether an agent’s language is truthful or merely persuasive. QUACK is released open-source so teams can run similar audits, train agents that optimize for groundedness, and extend the approach to other interactive, multimodal settings. open-source
Avoid common pitfallsLearn what failures to watch for
Credibility Assessment:
Authors include some with moderate h-index and affiliation to McGill (recognized institution), but arXiv venue and mostly junior authors — solid but not top-tier.