At a Glance
Most missing facts in on-site (air-gapped) AI agent systems are lost before the model ever sees them; fix the pipeline first instead of retraining or swapping models.
ON THIS PAGE
What They Found
Missing facts are usually caused by the surrounding software and tooling, not by the model alone. A nine-layer checklist shows how data can be deleted or hidden at ingestion, pagination, tool-handling, cache eviction, or orchestration steps. In a 75,476-trial sweep, deterministic pipeline failures (redaction, dropped pages, orchestrator truncation) explained most omissions, while model behavior contributed a smaller share. Practical runtime checks runtime checks (canary needles, coverage accounting, two-pass cross-checks) can detect many omissions in live, air-gapped deployments.
Not sure where to start?Get personalized recommendations
By the Numbers
1Overall omission rate observed: 62% of trials showed at least one missing canary.
2Deterministic pipeline layers (L0–L3: redaction, unfollowed pagination, orchestrator truncation, etc.) account for 68% of all observed omissions.
3Evaluation used a 75,476-trial sweep across five open-weight models and two inference engines to produce the reported attribution.
What This Means
Engineers running on-prem or air-gapped AI agents need this: it tells you where to instrument and what to fix when facts go missing. Technical leads and reliability teams can prioritize pipeline fixes (logging, pagination, tool integration) before investing in larger models or model tuning. Reliabilty-focused teams should consider the Hierarchical Multi-Agent Pattern when designing multi-agent trust and fault isolation.
Ready to evaluate your AI agents?
Learn how ReputAgent helps teams build trustworthy AI through systematic evaluation.
Learn MoreConsiderations
Results target single-box on-site deployments with 16–24 GB GPUs; behavior could differ on multi-GPU or CPU-only setups. Server-side engine settings (weight quantization, cache precision, positional encoding tweaks) were held constant here and may change omission patterns. A smaller 372-trial pilot covered common orchestration frameworks, so framework-level claims are preliminary and should be revalidated in your stack. For architecture-level guidance, consider the Orchestrator-Worker Pattern.
Methodology & More
Omission in on-site AI agents is often a pipeline problem: facts get lost before the model ever processes them. The work defines a nine-layer taxonomy spanning ingestion, tool protocol, caching, decoding, and orchestration, then instruments taps at each boundary. Each injected “needle” (a unique canary value) is tracked through encode/decode and logged at multiple checkpoints so exact-match presence or absence is measurable. Trials used three retrieval/attention architectures and two inference engines to separate engine-level behavior from pipeline effects. Findings show that deterministic, auditable software failures (like redaction, ignored pagination, or orchestrator truncation) account for the majority of missing facts—68% of omissions—while behavioral model layers explain the remainder. The study offers practical runtime detection techniques that leverage on-prem advantages (access to logits and telemetry): canary injection, coverage accounting, two-pass cross-checks, forced citation, and engine telemetry alarms. The takeaway for operators is concrete: add boundary logging, enforce pagination and tool-call robustness, and run simple runtime checks before assuming the model is at fault. For concrete patterns, see Dynamic Task Routing Pattern and Event-Driven Agent Pattern.
Avoid common pitfallsLearn what failures to watch for
Credibility Assessment:
Single author with very low h-index (2), no listed affiliation, and arXiv preprint with no citations — minimal identifiable credibility signals.