Robots That Only Speak When Useful: Faster, More Reliable Teamwork in the Home

In Brief

Calibrating language-based messages so robots only share answers they’re confident about makes multi-robot teams substantially more reliable and faster at finding and answering things in household scenes.

ON THIS PAGE

The Evidence

When robots use natural-language messages filtered by a confidence check, teammates avoid distracting or misleading tips and explore the environment more efficiently. A decentralized framework routes only high-confidence observations or answers, and those messages get turned into a shared map that guides navigation. Evaluated across hundreds of simulated home tasks, the method raises success rates and reduces time-to-answer compared with uncalibrated communication or no communication. Gains grow larger in bigger scenes and hold for small teams of two or three robots. The shared semantic value map is leveraged by the planner.

Not sure where to start?Get personalized recommendations

Learn More

Data Highlights

1Evaluation uses 420 question tasks across 70 realistic indoor scenarios (two-robot teams, three tasks per robot).

2A separate calibration set of 20 scenarios was used to fit the confidence filter for language outputs.

3Message-sending rates were tested from 0.25 to 4 messages per second to measure robustness to communication frequency.

Why It Matters

Engineers building multi-robot systems who need dependable teammate messages—so robots don’t waste time chasing hallucinated tips—will benefit from this approach. Technical leaders evaluating multi-agent reliability or agent-to-agent evaluation can use the calibration idea as a practical trust signal to reduce costly false leads. Researchers studying multi-agent communication can adopt the benchmark and calibration pipeline for reproducible comparisons.

Key Figures

Figure 1: In a household setting, robots exchange observations and reasoning to collaboratively complete their assigned tasks. Each agent generates confident and goal-directed messages using calibrated outputs from LLMs. The bottom-left image shows a bird’s-eye view of Robot 1’s navigation path after incorporating information received from Robot 2. The top-right sequence captures both robots’ camera views at different timestamps.

Fig 1: Figure 1: In a household setting, robots exchange observations and reasoning to collaboratively complete their assigned tasks. Each agent generates confident and goal-directed messages using calibrated outputs from LLMs. The bottom-left image shows a bird’s-eye view of Robot 1’s navigation path after incorporating information received from Robot 2. The top-right sequence captures both robots’ camera views at different timestamps.

Figure 2: An overview of our framework shows each robot with a perception module, a communication module, a planning module, and a confidence check module. At each time step, a robot generates local and global semantic values (SV) based on the current view and the communication message from the other robot. It navigates using a 2D weighted semantic value map and handles related object-check requests from other agents. The messages are generated based on the robot’s current view, which are calibrated by conformal prediction to enhance relevance.

Fig 2: Figure 2: An overview of our framework shows each robot with a perception module, a communication module, a planning module, and a confidence check module. At each time step, a robot generates local and global semantic values (SV) based on the current view and the communication message from the other robot. It navigates using a 2D weighted semantic value map and handles related object-check requests from other agents. The messages are generated based on the robot’s current view, which are calibrated by conformal prediction to enhance relevance.

Figure 3: The diagrams of SC vs. NTC on our MM-EQA dataset. (a)–(d) show the results for a 2-robot team. (a) The comparison between our method and baselines. (b) The ablative comparison of communication and conformal prediction modules. (c) The ablative comparison of the number of objects in the communication messages and the answering sharing mechanism. (d) The ablative comparison of baselines with our method at message-sending speeds of 0.25, 0.5, 1, 2, and 4 messages per second. (e) A scalability analysis comparing our method and baselines using a 3-robot team.

Fig 3: Figure 3: The diagrams of SC vs. NTC on our MM-EQA dataset. (a)–(d) show the results for a 2-robot team. (a) The comparison between our method and baselines. (b) The ablative comparison of communication and conformal prediction modules. (c) The ablative comparison of the number of objects in the communication messages and the answering sharing mechanism. (d) The ablative comparison of baselines with our method at message-sending speeds of 0.25, 0.5, 1, 2, and 4 messages per second. (e) A scalability analysis comparing our method and baselines using a 3-robot team.

$Figure 4: The comparisons of robot views and global SV maps among three methods. The red points represent the location of the target and the green points represent the position of the robot. Agents in the three methods start from the same pose. The question for this scenario is “Where is the red bear cushion?” For “Ours-No-CP” and “Ours”, Robot2 separately explores the same rooms at different times and sends messages to Robot1. The detailed messages are as follows: MSG 1 \text{MSG}_{1} : I see a basketboard, dolls, black chair that may be relevant to your target red bear cushion, and dolls may be your target at { p o s i t i o n 1 } \{position1\} . MSG 2 \text{MSG}_{2} : I see dolls that may be relevant to your target red bear cushion at { p o s i t i o n 2 } \{position2\} . MSG 3 \text{MSG}_{3} : I see bed, red pillow on blue chair that may be relevant to your target red bear cushion, and red pillow on blue chair may be your target at { p o s i t i o n 3 } \{position3\} . MSG 4 \text{MSG}_{4} : I see red pillow on blue chair that may be relevant to your target red bear cushion, and a red pillow on blue chair may be your target at { p o s i t i o n 4 } \{position4\} .$

Fig 4: Figure 4: The comparisons of robot views and global SV maps among three methods. The red points represent the location of the target and the green points represent the position of the robot. Agents in the three methods start from the same pose. The question for this scenario is “Where is the red bear cushion?” For “Ours-No-CP” and “Ours”, Robot2 separately explores the same rooms at different times and sends messages to Robot1. The detailed messages are as follows: MSG 1 \text{MSG}_{1} : I see a basketboard, dolls, black chair that may be relevant to your target red bear cushion, and dolls may be your target at { p o s i t i o n 1 } \{position1\} . MSG 2 \text{MSG}_{2} : I see dolls that may be relevant to your target red bear cushion at { p o s i t i o n 2 } \{position2\} . MSG 3 \text{MSG}_{3} : I see bed, red pillow on blue chair that may be relevant to your target red bear cushion, and red pillow on blue chair may be your target at { p o s i t i o n 3 } \{position3\} . MSG 4 \text{MSG}_{4} : I see red pillow on blue chair that may be relevant to your target red bear cushion, and a red pillow on blue chair may be your target at { p o s i t i o n 4 } \{position4\} .

Ready to evaluate your AI agents?

Learn how ReputAgent helps teams build trustworthy AI through systematic evaluation.

Learn More

Yes, But...

Results come from photo-realistic simulated homes, so real-world hardware, sensor noise, and network constraints may introduce new failure modes. Experiments focused on small teams (two to three robots); behavior and communication overhead for much larger teams remain untested. The approach depends on the quality of the vision-language detector and the calibration dataset size—poor perception or too little calibration data will weaken confidence estimates. For considerations of calibration workflow, teams may explore Human-in-the-Loop Pattern.

Deep Dive

CommCP is a decentralized communication framework that makes robot-to-robot natural-language messages useful by adding a statistical confidence filter. Each robot uses a vision-language detector to list observed objects, prompts a large language model to evaluate relevance to a partner’s request using a small set of discrete options, then converts the model’s token probabilities into a calibrated confidence via conformal prediction. Only messages that pass the confidence check are shared; when shared, those messages are projected into a shared semantic value map that the robot’s planner uses to bias exploration toward promising areas. The trust signal concept is central to this approach. The team created a multi-agent benchmark with 420 tasks across 70 scenes drawn from a photo-realistic 3D dataset and used a 20-scene calibration set for confidence fitting. In this setup, calibrated communication markedly increased task success and shortened exploration time compared to baselines that either never communicate or share raw, uncalibrated language outputs. Improvements were especially clear in larger scenes and when message rates varied. The approach provides a practical way to add a trust signal to agent-to-agent messages and can be integrated without central coordination, but further work is needed to validate on physical robots and to scale to large teams.

Explore evaluation patternsSee how to apply these findings

Learn More

Credibility Assessment:

Authors have low h-indices (≈2–3) and no affiliations or citations; signals an emerging/limited-information work.

multi-agent trust agent-to-agent evaluation agent reliability

Not sure where to start?