fase-semantic-entropy-code

name	fase-semantic-entropy-code
description	Fast Adaptive Semantic Entropy (FASE) methodology for quantifying uncertainty in multi-agent code generation. Approximates functional correctness via minimum spanning tree of structural and semantic dissimilarity graphs, achieving 25% improvement in Spearman correlation and 19% increase in ROCAUC over LLM-driven semantic entropy, at ~0.3% of the computational cost. Activation: semantic entropy, code uncertainty, multi-agent code quality, FASE, functional correctness estimation.
category	software-engineering
source	arxiv
arxiv_id	2606.09800
paper_title	FASE: Fast Adaptive Semantic Entropy for Code Quality
paper_authors	Shizhe Lin, Ladan Tahvildari
trigger	["semantic entropy code","FASE","fast adaptive semantic entropy","code quality uncertainty","multi-agent code generation","functional correctness estimation","code uncertainty quantification"]
version	1.0.0
created	2026-06-09

FASE: Fast Adaptive Semantic Entropy for Code Quality

Overview

FASE (Fast Adaptive Semantic Embedding) is a novel metric for quantifying uncertainty in multi-agent code generation workflows. It approximates functional correctness based on the minimum spanning tree (MST) of structural and semantic dissimilarity graphs, eliminating the need for costly LLM-driven equivalence checks used by traditional semantic entropy methods.

Key Results (arXiv:2606.09800):

• 25% average improvement in Spearman correlation vs. LLM-based semantic entropy
• 19% increase in ROCAUC score against Pass@1 from ground-truth test cases
• Requires only ~0.3% of the runtime cost of traditional semantic entropy approaches
• Evaluated on HumanEval and BigCodeBench using Qwen3-Embedding-8B model

Core Methodology

1. Structural Dissimilarity Graph

Build a graph where nodes are generated code samples and edges encode structural dissimilarity:

• Parse code into AST (Abstract Syntax Tree)
• Compute structural distance between ASTs (tree edit distance, node type distribution)
• Weight edges by structural dissimilarity

2. Semantic Dissimilarity Graph

Using embedding models (e.g., Qwen3-Embedding-8B):

• Embed each code sample into a semantic vector space
• Compute pairwise cosine distances between embeddings
• Weight edges by semantic dissimilarity

3. Minimum Spanning Tree (MST) Entropy

The key innovation:

• Combine structural and semantic graphs into a unified dissimilarity graph
• Compute the MST of this combined graph
• MST total weight serves as a proxy for semantic entropy:
  • Low MST weight: samples cluster tightly → high confidence in correctness
  • High MST weight: samples are scattered → high uncertainty, likely hallucination

4. Adaptive Thresholding

• Use the MST weight distribution to adaptively set confidence thresholds
• No ground-truth test cases required at inference time
• Calibrate thresholds on a small validation set

Implementation Steps

Step 1: Generate Multiple Code Samples

# Generate N code samples from the multi-agent system
samples = generate_code_samples(prompt, n=10)

Step 2: Build Structural Dissimilarity Matrix

 ast
 numpy  np

 ():
    
    tree_a = ast.parse(code_a)
    tree_b = ast.parse(code_b)
    
    features_a = extract_ast_features(tree_a)
    features_b = extract_ast_features(tree_b)
     np.linalg.norm(features_a - features_b)

 ():
    n = (samples)
    matrix = np.zeros((n, n))
     i  (n):
         j  (i+, n):
            d = ast_distance(samples[i], samples[j])
            matrix[i, j] = matrix[j, i] = d
     matrix