The Big Picture
Breaking the job into small, specialized AI agents (extractor, typer, verifier, curator) yields more accurate, more stable, and cheaper cyber threat knowledge graphs than using one large model.
ON THIS PAGE
The Evidence
Splitting the task into narrowly scoped agents and enforcing that each agent only uses evidence from the report reduces made-up facts and prevents errors from cascading. Semantic chunking preserves local meaning for long reports, and a verification and curation stage cleans up unsupported triples before graph assembly. The multi-agent setup Hierarchical Multi-Agent Pattern lets teams use smaller models while keeping or improving extraction quality and overall graph consistency. Full entity typing remains the hardest part and is the main source of remaining errors.
Data Highlights
1Aggregated corpus: over 230 human-annotated CTI reports used for training and evaluation.
2Extractor and typer training used up to 187 reports, producing 539 semantic chunks for model fine-tuning.
3Evaluation reports metrics as means over 3 runs with maximum standard deviation ≤ 1.0%, showing stable results across seeds.
What This Means
Security engineering teams and threat intelligence analysts who need structured, actionable threat graphs from vendor reports or open sources — they can get more accurate outputs while running smaller models. ML engineers and technical leaders evaluating LLM-based extraction should consider agent decomposition to control hallucinations, reduce cost, and enable clearer human-in-the-loop checks. For practitioners needing strategic context, see Multi-Agent Competitive Intelligence.
Not sure where to start?Get personalized recommendations
Key Figures
Fig 2: Figure 2 : TACTIC-KG framework.
Ready to evaluate your AI agents?
Learn how ReputAgent helps teams build trustworthy AI through systematic evaluation.
Learn MoreKeep in Mind
Results are demonstrated on public CTI reports and a MALOnt-aligned ontology; performance may change on proprietary or very different report styles. The framework improves extraction and graph quality, but full typing accuracy (correctly labeling both sides of a relation) remains a bottleneck. System quality still depends on ontology coverage and the quality of initial chunking and retrieval steps, and verification was trained on a smaller set (85 reports) which may limit generalization in edge cases. The note on initial steps can be seen in initial chunking and retrieval steps.
Methodology & More
TACTIC-KG reframes building a cyber threat intelligence knowledge graph as a sequence of small, specialized tasks handled by separate AI agents: an extractor finds candidate triples from sentence- or discourse-preserving chunks; a typer assigns ontology-compliant types; a verifier removes unsupported or low-confidence triples; and a curator merges and normalizes results into a consistent graph. The pipeline forces faithfulness by design: agents cannot add facts not grounded in the text, intermediate outputs are serialized for audit, and partial re-execution is supported so humans can inspect and correct stages. Evaluation used an aggregated benchmark of over 230 human-annotated CTI reports (including CTINEXUS and CTI-HAL), MALOnt for typing, and semantic matching for triplet alignment (threshold 0.6). Compared to a monolithic, prompt-driven baseline, the agentic setup produced more stable extraction quality and better graph-level consistency while enabling smaller backbone models and lower compute. Hybrid designs (specialized extractors/typers plus a shared verifier/curator) offered further gains. The main remaining challenge is improving full typing accuracy and extending robustness to noisy or adversarial inputs; future work aims at stronger ontology constraints and automated graph repair. For a broader perspective on coordination, see Market-Based Coordination Pattern and Evaluation-Driven Development (EDDOps).
Avoid common pitfallsLearn what failures to watch for
Credibility Assessment:
No affiliations listed, low author h-index (2) and arXiv-only with zero citations — limited signals of credibility.