In Brief
A multi-agent system produced graded, actionable exploit evidence for 67.8% of 603 disclosed vulnerabilities at about $1.50 each, and automatically created detection rules you can use to validate prioritization and build signatures.
ON THIS PAGE
Key Findings
Graded evidence is more useful than a yes/no proof: breaking exploitability into four depth levels reveals partial progress and gives signals suitable for both prioritization and detection. Generating small, targeted test apps that embed the actual vulnerable package avoids expensive project cloning and improves setup reliability. When an assessment reaches certain depth, telemetry from the exploit run yields high-quality Sigma and Snort rules. The system scales across languages and CWE types while keeping cost per case low. Dynamic Task Routing Pattern also supports flexible test-app generation across diverse environments. LLM-based oracle evaluations contribute structured verdicts with confidence scores.
Data Highlights
167.8% of 603 CVEs reached actionable evidence (L1+) — 409 out of 603 at $1.50 per CVE
270.3% (424/603) of CVEs succeeded in generation and deployment; of those, 96.5% reached L1+
3Detection rules derived from deeper (L2+) exploits showed stronger telemetry grounding (span-normalized effect, p = 0.035)
Implications
Security engineers and detection teams can use graduated exploit signals to validate or recalibrate vulnerability prioritization and to generate practical signatures from actual exploit traces. Human-in-the-Loop Pattern can help ensure prioritization aligns with real-world assessments. Platform and tooling teams can adopt targeted app generation to scale vulnerability testing without cloning entire projects, saving time and cost.
Explore evaluation patternsSee how to apply these findings
Key Figures
Fig 2: (a) Level distribution across 603 CVEs.
Fig 3: Figure 3: Exploitation level distribution for the 12 most frequent primary CWEs (% of CVEs in each row reaching each level). Cell shade is the within-CWE share of the L0–L3 column; rows marked with ∗ \ast are structurally capped at L2 by the oracle (§ 3.3 ).
Fig 4: (a) Span-normalized grounding by depth ( p = 0.035 p{=}0.035 ).
Ready to evaluate your AI agents?
Learn how ReputAgent helps teams build trustworthy AI through systematic evaluation.
Learn MoreYes, But...
Native memory-corruption vulnerabilities (low-level C/C++) were excluded, so results don’t apply to all bug classes. Generated test applications are minimal reproducers and may not reflect production defenses like web application firewalls or custom auth, so exploitability in the wild can differ. Dual-use risks exist because working exploit artifacts are produced, so access controls and responsible disclosure practices are necessary. For example, enforce Insecure Trust Boundaries checks as part of responsible handling.
Methodology & More
A five-agent pipeline runs graduated exploit assessments that produce three practical outputs: a depth score that captures how far an attack progressed, machine-readable detection rules derived from telemetry, and accumulated per-vulnerability-class knowledge to speed future tests. Instead of cloning original projects, the system generates minimal apps that import the vulnerable package version and intentionally expose behavior needed to show different exploit depths (for example: an endpoint that reflects errors for shallow evidence, seeded databases for data leaks, writable paths for file-write markers, and unsandboxed paths for code execution). An LLM-based oracle evaluates each tool action against vulnerability-specific criteria to produce structured verdicts with confidence scores and structural caps where full compromise is impossible. This pipeline aligns with an Orchestrator-Worker Pattern.
Avoid common pitfallsLearn what failures to watch for
Credibility Assessment:
Single author with low h-index (7), no clear institutional affiliation, arXiv preprint and no citations — signals of an emerging/limited profile.