swe-factory-benchmark-generation

name	swe-factory-benchmark-generation
title	SWE-Factory: Your Automated Factory for Issue Resolution Training Data and Evaluation Benchmarks
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2506.10954
keywords	["automated dataset generation","GitHub issues","software engineering","benchmarks","fail2pass validation"]
description	Automate construction of GitHub issue resolution benchmarks via multi-agent LLM coordination, binary file recovery, and exit-code-based validation, achieving 337 valid instances at $0.047 per instance.

SWE-Factory: Automated Benchmark Generation

Core Concept

SWE-Factory fully automates GitHub issue resolution dataset construction through a multi-agent LLM system that coordinates environment setup, binary file recovery, and fail2pass validation. Instead of manual inspection and custom log parsing, the system uses exit-code-based testing (standardized Unix return codes), successfully generating 337 valid benchmarks from 671 issues across Python, Java, JavaScript, and TypeScript at $0.047 per instance and 96% quality.

Architecture Overview

• SWE-Builder Multi-Agent System: Four collaborative agents (Repository Explorer, Environment Manager, Test Manager, Test Analyst) with shared Environment Memory Pool for configuration reuse
• Binary File Recovery: Automatically downloads missing test binaries and removes incomplete hunks from patches
• Exit-Code-Based Validation: Standardized fail2pass testing via Unix exit codes ($rc) instead of manual log parsing, achieving F1=0.99
• Environment Memory Pool: Caches successful Docker configurations from similar repository versions, reducing redundant iterations
• Language Support: Python, Java, JavaScript, TypeScript with unified testing interface

Implementation

Step 1: Binary File Recovery System

import os
import subprocess
from pathlib import Path

class BinaryFileRecoverer:
    """
    Automatically downloads missing binary test files from repositories
    and removes incomplete binary hunks from patches.
    """

    def __init__(self, repo_path, repo_url):
        self.repo_path = repo_path
        self.repo_url = repo_url

    def identify_missing_binaries(self, patch_content):
        """
        Parse patch file to identify missing binary files.
        Binary patches show "Binary files ... differ" marker.
        """
        missing_binaries = []
        lines = patch_content.split('\n')

        for i, line  (lines):
               line    line:
                
                 j  (i-, (, i-), -):
                     lines[j].startswith():
                        filename = lines[j].split()[].replace(, )
                        missing_binaries.append(filename)
                        

         missing_binaries

     ():
        
        downloaded = []

         filename  filenames:
            
            raw_url = .repo_url.replace(, )
            raw_url = 

            :
                result = subprocess.run(
                    [, , , filename, raw_url],
                    cwd=.repo_path,
                    timeout=,
                    capture_output=
                )

                 result.returncode ==   os.path.exists(os.path.join(.repo_path, filename)):
                    downloaded.append(filename)
                    ()
             Exception  e:
                ()

         downloaded

     ():
        
        lines = patch_content.split()
        cleaned = []
        skip_binary_section = 

         i, line  (lines):
               line    line:
                skip_binary_section = 
                cleaned.append(line)  
             line.startswith()  line.startswith():
                skip_binary_section = 
                cleaned.append(line)
              skip_binary_section  line.startswith((, )):
                cleaned.append(line)

         .join(cleaned)