Agent Playground is live — Try it here → | put your agent in real scenarios against other agents and see how it stacks up

Key Takeaway

Targeted, low-cost clarification questions grounded in specific video moments let camera-based assistants identify the user’s intended event, producing more correct, verifiable answers and fewer repeated mistakes.

What They Found

Represent competing meanings explicitly as grounded candidate events (when and where something happened) and coordinate three specialized agents to (1) find candidates in the video, (2) pick cost-aware clarification questions, and (3) answer using the selected evidence. Bounding the number and complexity of questions keeps user effort low while still resolving many ambiguities. Learning from verified feedback builds intent-specific skills that reduce repeated misinterpretations and improve later responses. The biggest gains appear when competing referents occur at different times in the video, rather than multiple things visible at once.

By the Numbers

13 specialized agents coordinate grounding, clarification, and answering to resolve intent.
2Conditional answers are limited to at most 2 interpretations to keep responses compact and actionable.
3Grounding correctness requires both temporal and spatial overlap ≥ 0.5 (50% IoU) when spatial evidence exists.

Implications

Engineers building camera-based assistants and interactive agents — to reduce wrong answers that are hard for users to detect and to minimize interruptions. Product and research leads evaluating agent reliability and user burden — to balance clarification cost against answer quality and to reduce recurring failures via simple learned skills.
Explore evaluation patternsSee how to apply these findings
Learn More

Key Figures

Figure 1: Different encounters can support different interpretations of the same question. ReferClaw asks for clarification to identify the intended encounter and select evidence for answering.
Fig 1: Figure 1: Different encounters can support different interpretations of the same question. ReferClaw asks for clarification to identify the intended encounter and select evidence for answering.
Figure 2: Overview of ReferClaw. The system coordinates three agents (IGA, CCA, EFA) to respectively ground candidate interpretations, requests clarification when useful, and verifies answers against the resolved intent and evidence. A fixed controller bounds interaction. Top: verified feedback updates prompt-level skills immediately and agent-specific LoRA adapters during idle periods.
Fig 2: Figure 2: Overview of ReferClaw. The system coordinates three agents (IGA, CCA, EFA) to respectively ground candidate interpretations, requests clarification when useful, and verifies answers against the resolved intent and evidence. A fixed controller bounds interaction. Top: verified feedback updates prompt-level skills immediately and agent-specific LoRA adapters during idle periods.
Figure 8: Which core design helps which case. Rows are the four core designs of ReferClaw; columns are the case types defined by the annotations (diagnostic groups G1–G5, well-specified questions, questions that require state or action reasoning after the referent is resolved, and failure types that recur along the interaction stream). Cell shading gives the qualitative level of help, and cell notes quote the result that supports it.
Fig 5: Figure 8: Which core design helps which case. Rows are the four core designs of ReferClaw; columns are the case types defined by the annotations (diagnostic groups G1–G5, well-specified questions, questions that require state or action reasoning after the referent is resolved, and failure types that recur along the interaction stream). Cell shading gives the qualitative level of help, and cell notes quote the result that supports it.
Figure 9: Qualitative comparison on a two-question episode. z 1 z_{1} , z 2 z_{2} : grounded candidates (dashed boxes: their tracks); tags mark the core design behind each ReferClaw step (D1–D4 as in Fig. 8 ); ✓/ × \times : correct or wrong for the target intent z 2 z_{2} .
Fig 6: Figure 9: Qualitative comparison on a two-question episode. z 1 z_{1} , z 2 z_{2} : grounded candidates (dashed boxes: their tracks); tags mark the core design behind each ReferClaw step (D1–D4 as in Fig. 8 ); ✓/ × \times : correct or wrong for the target intent z 2 z_{2} .

Ready to evaluate your AI agents?

Learn how ReputAgent helps teams build trustworthy AI through systematic evaluation.

Learn More

Considerations

ReferClaw is a research prototype evaluated on augmented benchmarks and simulated respondents; results may differ in live, noisy deployments. Clarification still imposes effort and is bounded, so some complex ambiguities may remain unresolved under the fixed question budget. Safety-critical use would require additional validation, user testing, and privacy safeguards before deployment. Spiraling Hallucination Loops

Deep Dive

Everyday voice or short-phrase requests to a camera-based assistant can map to multiple events in a video (for example, “Did she put it back?” could refer to different people, objects, or times). Rather than guessing, build a small set of explicit candidate interpretations by rewriting the question to name possible referents and attaching the video moments (temporal intervals and object tracks) that support each one. A three-agent system then uses those shared candidates: an intention grounding agent finds and scores candidates, a cost-aware clarification agent proposes short questions that trade off information gain against user effort, and an evidence-fused answering agent produces answers tied to the chosen interpretation. A fixed controller limits how many and how complex clarification turns can be, and verified feedback after each interaction is used to create intent-specific skills and agent-level updates so the system makes fewer of the same mistakes later. Evaluated on two augmented egocentric benchmarks and two image-based clarification sets, the system asks more useful clarifying questions than interactive baselines, improves intent-aligned and jointly grounded answers while keeping user cost low, and yields the largest gains when competing referents are separated in time. Learned skills also improve performance in later interactions and the frozen system transfers to image benchmarks without degrading well-specified answers. three-agent system evidence-fused answering
Explore evaluation patternsSee how to apply these findings
Learn More
Credibility Assessment:

Contains Angela Yao (h-index 43 >40), indicating a top researcher despite arXiv venue—rates as high credibility.