At a Glance
When compared to size-matched human templates, AI-generated infrastructure-as-code has about 3.2–3.9 times the density of scanner-flagged security issues per declared resource.
ON THIS PAGE
What They Found
AI models produce infrastructure templates with substantially more scanner findings per declared resource than humans, even after matching for file size. That gap is consistent across cloud providers, formats, and model families, and is largest on the simplest scenarios. Vendor-supplied "extended thinking" reasoning gives a modest improvement, while prompted chain-of-thought provides no measurable security benefit because the mechanism is barely used when generating infrastructure code. Chain-of-Thought Pattern.
By the Numbers
1Generated artifacts hit a 99.7% yield (1,196 of 1,200 requested configurations were produced).
2Scanners reported 38,803 findings in total (Checkov 14,017; KICS 14,033; Trivy 10,753).
3AI-generated templates show a vulnerability-density multiplier of roughly 3.21×–3.87× versus a size-matched human baseline.
What This Means
Cloud engineers and SREs who plan to use AI assistants for writing deployment templates need to treat model output as higher-risk by default and add review or automated safeguards. Security teams and technical leaders evaluating AI adoption should require size-matched comparisons and multi-engine scanning before trusting model-generated infrastructure. size-matched comparisons
Not sure where to start?Get personalized recommendations
Key Figures
Fig 1: Fig. 1: Density distributions by configuration. The right skew motivates rank-based inference.
Fig 2: Fig. 2: Vulnerability density declines with artifact size in both corpora, requiring size-matched comparison.
Fig 3: Fig. 3: Size-matched comparison against the human baseline. ∗ p < 0.05 {}^{*}p<0.05 , p ∗ ∗ < 0.01 {}^{**}p<0.01 , ∗ ∗ ∗ p < 0.001 {}^{***}p<0.001 (Mann–Whitney U U ).
Fig 4: Fig. 4: Per-configuration density relative to the human baseline. Purple: vendor extended thinking. Green: prompted chain-of-thought.
Ready to evaluate your AI agents?
Learn how ReputAgent helps teams build trustworthy AI through systematic evaluation.
Learn MoreConsiderations
The human baseline is not a perfect control—those templates were not written against the same scenarios and may be curated examples rather than production code. Scanner findings are policy deviations flagged by static tools, not a direct measure of exploitability or real-world attacks. Results reflect the specific models, formats, and scanner rules used; hosted models and scanner coverage change over time and across IaC languages. Hallucination Propagation
Methodology & More
A 100-scenario benchmark (60 simple, 40 complex) asked models to generate infrastructure-as-code across major cloud platforms and Kubernetes without any security instructions, then scanned outputs with three popular static engines. A size-matched human corpus (634 templates) was scanned through the same toolchain so findings could be compared on a per-declared-resource basis rather than by raw file counts. That size-matched comparison reveals a consistent gap: generated templates carry about 3.2–3.9 times the scanner-flagged issue density of human-authored templates. The gap holds across providers, formats, and model configurations, suggesting an upstream cause such as training data biases toward permissive defaults. Vendor "extended thinking" APIs produced a small but real reduction in findings; prompting for chain-of-thought did not help, largely because the reasoning mechanism contributed under 1% of output tokens for these tasks. The study also documents practical pitfalls: multi-engine pipelines can silently lose or miscount findings, scanners report policy issues not exploitability, and hosted models evolve—so teams should release data and use size-matched human baselines, multi-engine scans, and exploitability triage before adopting generated IaC in production. Orchestrator-Worker Pattern Ground Truth
Avoid common pitfallsLearn what failures to watch for
Credibility Assessment:
Single author with low h-index (3), no listed affiliation, arXiv preprint and no citations — fits emerging/limited-info category.