adversarial-code-verification

name	adversarial-code-verification
title	Rethinking Verification for LLM Code Generation: From Generation to Testing
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2507.06920
keywords	["Code Verification","Test Generation","Adversarial Testing","Code Generation Evaluation","Program Verification"]
description	Generate adversarial test suites that catch LLM code errors by analyzing constraint differences between correct and incorrect solutions. SAGA framework improves test detection rate 9.55% and verifier accuracy 12.14% by using human bug patterns and multi-input differential analysis.

Adversarial Code Verification: Generating Discriminative Test Cases from Solution Patterns

Current benchmarks for LLM code generation use limited, homogeneous test suites that inflate performance metrics and fail to catch real errors. A model scoring 95% on standard benchmarks may fail 50% of real submissions because test suites are too narrow, allowing solutions with subtle bugs to pass. SAGA solves this through strategic analysis: first, extract constraint insights from correct solutions to identify edge cases; second, analyze patterns in human bug submissions to create targeted test cases; third, generate diverse inputs exploring different error classes. The result is test suites that reliably distinguish correct from incorrect code, giving RL systems honest reward signals.

When training code generation models with RL, weak test suites corrupt the reward signal—models learn to pass bad evaluations rather than produce correct code. SAGA test suites catch bugs that simple unit tests miss: off-by-one errors in loops, incorrect boundary handling, type confusion, and logic inversions. The human-LLM collaborative approach combines human insight about what makes problems hard with LLM efficiency at generating diverse test cases.

Core Concept

SAGA generates adversarial tests through three coordinated phases. First, multidimensional analysis examines correct solutions to extract constraint patterns: what boundary conditions do they handle, what invariants do they maintain? Second, differential analysis compares failed solutions with corrected versions to identify common error patterns—these patterns become targets for test generation. Third, dual-input strategy combines both ground-truth and buggy solutions to produce tests that distinguish right from wrong. The framework introduces new evaluation metrics beyond simple accuracy: detection rate (what proportion of known bugs does the test suite find), verifier accuracy (does the suite correctly identify all incorrect solutions), and distinct error pattern coverage (breadth of error types detected).

Architecture Overview

• Multidimensional Analyzer: Extracts constraint patterns, boundary conditions, and invariants from correct solutions
• Differential Analyzer: Compares correct vs. buggy submissions to identify error patterns
• Constraint-Differential Test Generator: Creates test cases targeting specific constraint violations and error patterns
• Test Discriminator: Validates that generated tests actually distinguish correct from incorrect code
• Metrics Engine: Computes detection rate, verifier accuracy, and error pattern coverage
• Human-LLM Collaborative Pipeline: Combines human problem insight with LLM test generation at scale

Implementation

This example demonstrates extracting constraint patterns from correct solutions to guide test generation.