At a Glance
FASE spots when AI-generated code is likely wrong without needing test suites by clustering code meaning (embeddings), giving a fast, useful signal for multi-agent workflows.
ON THIS PAGE
Key Findings
Embedding models can replace expensive pairwise checks by large language models to estimate how functionally consistent a set of code samples is. By building a graph from embedding distances, extracting a minimum spanning tree, and using an adaptive clustering threshold, FASE computes a semantic entropy that correlates with functional correctness. The method is far cheaper to run and works across multiple code models and embedding encoders, making it practical for monitoring many agents in a pipeline. Embedding models for evaluation and graph-based approaches like minimum spanning tree provide a scalable alternative.
Data Highlights
1Benchmarked on 164 HumanEval tasks and 148 BigCodeBench-hard tasks to test generality across simple and harder problems
2Evaluated with 4 open-source 7B parameter code models and 4 different encoder-only embedding models to measure robustness
3Generated 10 code samples per task (per model) and computed pairwise embedding distances to form the entropy estimate
Why It Matters
Engineers building multi-agent code systems who need a cheap way to flag unreliable agent outputs before they cascade. Technical leads and platform teams who must monitor agent reliability and trust can use FASE to triage which code artifacts need deeper testing or human review. Multi-agent QA testing can benefit from quick reliability signals.
Not sure where to start?Get personalized recommendations
Key Figures
Fig 1: Figure 1 . The risk of hallucination from unreliable agents (red) undermines the credibility of coherent agents (blue) and propagates across different workflow architectures.
Fig 2: Figure 2 . The workflow of computing encoder-based semantic entropy for the samples of code generated for a given task.
Fig 3: Figure 3 . Pass@1 distribution of code generated by Mistral, CodeLlama, DeepseekCoder, Qwen2.5-Coder on HumanEval and BigCodeBench-hard
Fig 4: (a) All-MiniLM PDM
Ready to evaluate your AI agents?
Learn how ReputAgent helps teams build trustworthy AI through systematic evaluation.
Learn MoreYes, But...
Embeddings can miss small but critical code changes (a single operator flip can change behavior), so FASE is best paired with structural checks or targeted tests for high-risk code. FASE may produce low entropy when all generated solutions are similarly wrong; it flags disagreement, not absolute correctness. Experiments used only 7B models and two benchmarks, so performance may vary with larger models, different tasks, or different prompt styles. For a focus on risk, consider Capability Discovery Pattern.
The Details
FASE replaces costly large-model equivalence checks with a light-weight, embedding-driven pipeline to estimate whether multiple AI-generated implementations of the same task are functionally consistent. For each set of generated solutions, FASE computes embedding vectors, measures pairwise cosine distances, and builds a compact graph via the minimum spanning tree to highlight the most informative relationships. An adaptive density clustering threshold is chosen from the distribution of MST edge weights (using a mode estimate scaled by mean distances) to form equivalence groups; semantic entropy is then computed over those groups to quantify uncertainty about functionality. In experiments on standard code generation benchmarks and across four 7B code models and four embedding encoders, FASE produced a useful signal that correlates with functional correctness while dramatically lowering computation compared with bidirectional checks from large models. Practically, FASE is best used as a fast monitor inside multi-agent pipelines: it can flag tasks that need extra verification, guide agent selection, or reduce wasted compute on full-fledged verification for clearly consistent outputs. For higher assurance, combine FASE with structural analysis, focused tests, or alignment checks to catch consistent-but-wrong solution sets. Multi-agent pipelines.
Test your agentsValidate against real scenarios
Credibility Assessment:
ArXiv preprint with no affiliations and zero citations. Two authors (no clear institutional or high-h-index signals), so classified as emerging/limited information rather than unknown.