hardtests-code-verification

name	hardtests-code-verification
title	HardTests: Synthesizing High-Quality Test Cases for LLM Coding
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2505.24098
keywords	["Test Synthesis","Code Verification","LLM Reasoning","Edge Cases"]
description	Generate comprehensive test cases for code problems that reliably detect wrong solutions through LLM-based edge case synthesis and test quality ranking.

Synthesize Edge-Case Tests That Catch Wrong Solutions

Verifying code correctness is critical for LLM post-training, but automated test generation struggles with hard problems where wrong solutions are cleverly disguised. HardTests introduces HARDTESTGEN: an LLM-based pipeline that synthesizes high-quality test cases by actively discovering edge cases that reveal subtle bugs. Unlike simple random testing, this approach systematically explores the solution space to find failure-inducing inputs.

The key insight is that humans write effective tests by thinking about edge cases—boundary conditions, off-by-one errors, special cases. LLMs can emulate this reasoning by being prompted to think adversarially: "What inputs would break incorrect implementations?" By generating many candidate tests and ranking them by their ability to discriminate between correct and wrong solutions, the pipeline builds powerful test suites.

Core Concept

HARDTESTGEN works in three phases:

• Test generation: Use LLM to generate diverse test cases, explicitly including edge cases and boundary conditions
• Test discrimination: Filter tests by their ability to distinguish correct from incorrect solutions
• Test diversity: Ensure tests cover different failure modes and input types
• Quality ranking: Rank tests by discriminative power across multiple solution variants

The framework recognizes that correct solutions often fail on specific edge cases while buggy solutions fail elsewhere. By finding tests that separate these failure patterns, you create a verifier that catches disguised wrong answers.

Architecture Overview

• Problem analysis module: Extract key problem characteristics (constraints, edge cases)
• Candidate test generator: Create diverse test cases including edge cases, boundaries, and random inputs
• Solution evaluator: Run tests against correct and incorrect reference solutions
• Discriminator function: Measure test quality by how many wrong solutions it rejects
• Diversity optimizer: Select diverse test set rather than redundant hard tests
• Ranking system: Prioritize tests by their utility for verification

Implementation

Build a test synthesis pipeline that generates and ranks test cases:

# HardTests: Test synthesis for LLM code verification
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from typing import List, Tuple, Set

class HardTestGenerator:
    def __init__(self, model_name="gpt-3.5-turbo"):
        self.model = AutoModelForCausalLM.from_pretrained(model_name)
        self.tokenizer = AutoTokenizer.from_pretrained(model_name)

    def generate_candidate_tests(self, problem_description: str, num_candidates: int = 20) -> List[str]:
        """
        Generate diverse test cases by prompting LLM to think about edge cases.
        """
        prompt = f"""You are a software engineer creating comprehensive tests for a coding problem.
Problem: {problem_description}

Generate {num_candidates} test cases that would catch incorrect solutions. Include:
1. Basic valid inputs
2. Edge cases (empty, single element, maximum/minimum values)
3. Boundary conditions
4. Off-by-one errors
5. Special cases (zero, negative, duplicates)

For each test, provide:
- Input specification
- Expected output
- Why this test matters

Generate test cases:"""

        # Generate multiple test scenarios
        response = self.model.generate(
            self.tokenizer.encode(prompt, return_tensors='pt'),
            max_length=2000,
            temperature=0.8,
            num_return_sequences=1
        )
        test_text = self.tokenizer.decode(response[])

        
        tests = ._parse_test_cases(test_text)
         tests[:num_candidates]

     () -> []:
        
        tests = []
        lines = test_text.split()
        current_test = []

         line  lines:
             line.strip().startswith()  line.strip().startswith():
                 current_test:
                    tests.append(.join(current_test))
                current_test = [line]
             current_test:
                current_test.append(line)

         current_test:
            tests.append(.join(current_test))

         tests

     () -> :
        
        correct_passed = 
        wrong_failed = 

        
         sol  correct_solutions:
            :
                 ._test_solution(sol, test):
                    correct_passed += 
            :
                  

        
         sol  wrong_solutions:
            :
                  ._test_solution(sol, test):
                    wrong_failed += 
            :
                wrong_failed +=   

        correct_pass_rate = correct_passed / (correct_solutions)  correct_solutions  
        wrong_fail_rate = wrong_failed / (wrong_solutions)  wrong_solutions  

        
        discrimination_score = wrong_fail_rate - ( - correct_pass_rate)
         (, discrimination_score)

     () -> :
        
        
        
        
        
        
        :
            
            exec_globals = {}
            (solution_code, exec_globals)

            
            
            test_input, expected_output = ._parse_test_spec(test_spec)
            result = exec_globals[](test_input)
             result == expected_output
         Exception  e:
             

     () -> :
        
        
        
         , 

     () -> []:
        
        
        candidates = .generate_candidate_tests(problem, num_candidates=)

        
        
        correct_solutions = [._get_reference_solution(problem)]

        
        wrong_solutions = ._generate_wrong_solutions(problem, num_wrong=)

        
        scored_tests = []
         test  candidates:
            score = .rank_test_by_discrimination(test, correct_solutions, wrong_solutions)
            scored_tests.append((test, score))

        
        scored_tests.sort(key= x: x[], reverse=)

        
        selected = ._select_diverse_subset(scored_tests, num_tests)

         [t[]  t  selected]

     () -> []:
        
        prompt = 

        
        
         [, ]

     () -> :
        
        
         

     () -> [[, ]]:
        
        selected = []
        remaining = scored_tests.copy()

        
         (selected) < k  remaining:
            
            best = remaining.pop()
            selected.append(best)

            
            remaining = [t  t  remaining   ._are_similar_tests(best[], t[])]

         selected[:k]

     () -> :
        
        
        common_tokens = ((test1.split()) & (test2.split()))
         common_tokens > ((test1.split())) * 

Implement a verification utility for code solutions using the generated tests:

def verify_solution_with_hard_tests(solution_code: str, test_suite: List[str]) -> dict:
    """
    Verify a code solution against high-quality test suite.
    Returns detailed results about which tests pass/fail.
    """
    results = {
        'passes': 0,
        'fails': 0,
        'errors': 0,
        'failed_tests': [],
        'error_tests': []
    }

    for test_idx, test_spec in enumerate(test_suite):
        try:
            # Execute test
            if test_passes(solution_code, test_spec):
                results['passes'] += 1
            else:
                results['fails'] += 1
                results['failed_tests'].append((test_idx, test_spec))
        except Exception as e:
            results['errors'] += 1
            results['error_tests'].append((test_idx, str(e)))

    results['score'] = results['passes'] / len(test_suite)
    return results

def test_passes(solution_code: str, test_spec: str) -> bool:
    """Execute a test specification against solution code"""
    
    :
        
        exec_context = {}
        (solution_code, exec_context)
        
           
    :
         

Practical Guidance

Aspect	Recommendation	Notes
Candidate generation	3-5x needed tests	Generate excess, filter by quality
Test discrimination threshold	0.5+ score	Tests should fail >50% wrong solutions
Reference solutions	1 correct + 5-10 wrong	Balance verification cost with ranking quality
Test diversity	Ensure coverage of ≥3 failure modes	Avoid redundant tests
Test format	Structured input/output pairs	Enables automated execution

When to use HardTests:

• Verifying complex coding problems with subtle edge cases
• Post-training LLMs with RL/DPO on code tasks
• Want to catch disguised wrong solutions (clever bugs)
• Building robust test suites for code competitions
• Need high-confidence verification without human review

When NOT to use:

• Simple problems with obvious correct/wrong answers
• Test suites already exist and are comprehensive
• Edge cases don't matter (prototyping, tutorials)
• Testing speed is more important than accuracy
• Problem correctness is hard to formalize (creative coding)

Common pitfalls:

• Generating only obvious test cases (boundary value testing sufficient)
• Not ranking tests by discrimination ability
• Too few reference wrong solutions for effective ranking
• Tests that are too similar (redundant coverage)
• Overrelying on LLM-generated wrong solutions without validation
• Not considering solution variability (different approaches, languages)

Reference

HardTests: Synthesizing High-Quality Test Cases for LLM Coding https://arxiv.org/abs/2505.24098