The Big Picture
Routing only the right signals and distilled operational knowledge to a shared language model—via a flexible Skill layer—cut alert volume by 75%, removed most non-actionable alerts, and reduced time-to-fix by over 50%.
ON THIS PAGE
The Evidence
A three-role agent setup (release watcher, proactive inspector, and alert diagnostician) covers most day-to-day operations by selecting relevant data and domain knowledge before reasoning. A Semantic Capability Matching Pattern layer automatically assembles the right logs, metrics, and contextual rules per business-module and evolves through plain-language feedback from engineers. A single feedback signal both updates the Skill that routes data and distills case memory into reusable knowledge, so the system improves jointly without separate update pipelines.
Not sure where to start?Get personalized recommendations
Data Highlights
175% reduction in fired alerts after deployment
2≈95% reduction in practitioner-facing non-actionable alerts
380% accuracy on root-cause analysis for actionable alerts, with over 50% reduction in mean time to resolution
What This Means
Site reliability engineers, platform leads, and teams building AI-driven operations should pay attention—this approach directly reduces alert fatigue and speeds up troubleshooting without heavy manual wiring. AI/ops engineers can use the Flexible Skill design to scale automated checks and keep human oversight where it matters.
Key Figures
Fig 1: Figure 1 : Overview of the Bian Que architecture. Operational events from the OPS platform (top) are dispatched to a matching Agent, which invokes one or more matched Skills to assemble the relevant data (system signals: logs, metrics, change events) and knowledge (domain knowledge distilled from case memory, seeded by operational handbooks) for the LLM to reason over; the resulting diagnosis is returned to the OPS platform. Practitioner feedback flows back along two parallel pathways (yellow: Skill refinement; purple: memory-to-knowledge distillation).
Fig 2: Figure 2 : Agent Matrix and Skill Pool. Each Agent (top) implements one canonical pattern; Skills (middle) are business-module orchestration units in a shared pool. A single execution composes one Agent with matched Skills, which call into the knowledge store and data sources (bottom).
Fig 3: Figure 3 : Flexible Skill lifecycle. New Skills are generated from seed configurations with validation and retry. Existing Skills are updated via practitioner feedback through a natural-language interface.
Ready to evaluate your AI agents?
Learn how ReputAgent helps teams build trustworthy AI through systematic evaluation.
Learn MoreConsiderations
Results come from a single large-scale deployment on one search engine, so expect variation when applying to different stacks or business logic. The approach assumes an existing, mature data stack and a capable language model (models above ~30B were reported to work well). Guardrails Pattern Human-in-the-loop supervision and careful onboarding of business/module descriptors remain necessary to avoid missed or incorrect automated actions.
Methodology & More
Large-scale online services generate vast, heterogeneous signals—metrics, logs, traces and deployment events—while operational knowledge (handbooks, past cases) is noisy and moves with the system. The framework organizes operational work into three practical roles: intercepting risky releases, running scheduled proactive checks, and diagnosing alerts that do surface. Instead of dumping all signals into the model, a Flexible Skill layer maps an event and its business-module context to the precise subset of data and domain knowledge needed for reasoning. Skills are structured artifacts (stored as YAML) that describe which data sources and distilled knowledge to surface for a given scenario. Chain of Thought Pattern Skills are generated and refined by the same language model used for reasoning, and engineers can correct or update Skills via natural-language instructions. A unified feedback loop turns practitioner corrections into two things at once: (1) targeted Skill updates that change what data and knowledge get assembled next time, and (2) memory-to-knowledge distillation that turns solved cases into reusable domain knowledge for future reasoning. Deployed on a massive e-commerce search engine, the system reduced alert volume by 75%, cut practitioner-facing noise by about 95%, achieved roughly 80% root-cause accuracy on alerts that required attention, and shortened mean time to resolution by over 50%. The design is model-agnostic above a practical scale and intended to generalize by seeding Skills from existing data-lake descriptors, keeping per-business onboarding effort focused on those descriptors rather than framework changes. Orchestrator-Worker Pattern
Avoid common pitfallsLearn what failures to watch for
Credibility Assessment:
ArXiv preprint with multiple authors showing low h-indexes (mostly 2–5) and no listed top institutions or venue — emerging/limited-info.