Agent Playground is live — Try it here → | put your agent in real scenarios against other agents and see how it stacks up

Key Takeaway

A privacy-first multi-agent setup lets non-technical utility staff query industrial smart meter data in plain language using local open models, and yields clear accuracy gains on resource‑constrained hardware.

The Evidence

A three-agent pipeline (orchestrator, coding agent, interpretation agent) running open models on-premise enables natural‑language queries over real smart meter databases without sending data to cloud services. Breaking the task into agents and adding an automated self-correction loop meaningfully improves executable query success, especially on smaller local models. Injecting a curated, per-query domain knowledge base improves semantic correctness but can slightly reduce raw executability in some setups. On very large models, single-agent baselines sometimes match or beat full orchestration, so orchestration should be applied selectively.

Data Highlights

1Full multi-agent on local hardware increased execution recall to 42.4% and delivered a +6.1 percentage‑point gain in execution recall over the strongest single‑agent baseline.
2Adding self‑correction raised executable success rates by up to 20.0 percentage points in the multi‑agent track.
3On the largest model tier, a single‑agent baseline reached 54.7% execution recall while multi‑agent scenarios showed executable success rates near or above 94.7%, highlighting that automated metrics can undercount practically useful outputs.

What This Means

Engineers building conversational front ends for industrial telemetry and technical leaders in utilities who must keep sensitive consumer data on‑premise will benefit most. Product teams evaluating agent orchestration tradeoffs can use the results to decide when to enable multi‑agent decomposition, self‑correction, or domain knowledge injection depending on model size and hardware limits.
Explore evaluation patternsSee how to apply these findings
Learn More

Ready to evaluate your AI agents?

Learn how ReputAgent helps teams build trustworthy AI through systematic evaluation.

Learn More

Considerations

Evaluation used a single industrial deployment and 62 verified natural‑language to SQL examples, so results may vary across different operators, geographies, or schema styles. Experiments ran zero‑shot prompts on one model family; few‑shot examples, agent fine‑tuning, or other open models could change outcomes. Automated metrics tended to undervalue useful but structurally different SQL; include human review when judging practical utility.

The Details

OpenMeter is a privacy‑first system that lets non‑technical utility staff ask questions of industrial smart meter databases in plain English while keeping all data and model runs inside the operator’s controlled infrastructure. The pipeline decomposes work across three specialized agents: an orchestrator that classifies intent, curates a small per‑query slice of domain rules and builds an execution plan; a coding agent that generates and executes SQL (and visualizations) in a sandbox with an autonomous self‑correction loop for execution errors; and an interpretation agent that turns numeric results into human‑readable summaries, flags anomalies by rule ID, and uses cautious language for causal claims. The orchestrator’s dynamic filtering of a versioned domain knowledge base avoids heavy external retrieval stacks and keeps prompts compact.
Need expert guidance?We can help implement this
Learn More
Credibility Assessment:

Authors have low h‑index and no clear institutional signals; arXiv preprint with no citations suggests emerging/limited credibility.