菜单
Automatically generate executable software engineering environments from GitHub pull requests at million-scale using an autonomous building agent. Detects superficial verification patterns to force genuine code execution testing.
默认使用会先检查来源的 Prompt；你也可以切换为直接命令，或下载本地副本。
决定是否安装前，请先阅读 SKILL.md，以及 SkillsMP 当前展示的配套文件。
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
直接命令不会经过审查 Prompt；运行前请先检查来源。
npx skills add https://github.com/ADu2021/skillXiv --skill swe-universe-environments命令会保持在同一行。复制前请横向滚动并检查完整内容。
想先保存到本地？可下载 SkillsMP 当前能够提供的文件。
基于 SOC 职业分类
正在显示 SKILL.md
| name | swe-universe-environments |
| title | SWE-Universe: Scale Real-World Verifiable Environments to Millions |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2602.02361 |
| keywords | ["Software Engineering Agents","Environment Generation","Scalable Verification","GitHub","Autonomous Building"] |
| description | Automatically generate executable software engineering environments from GitHub pull requests at million-scale using an autonomous building agent. Detects superficial verification patterns to force genuine code execution testing. |
Software engineering (SWE) task datasets are small and limited to curated repositories. Agent training requires diverse, real-world verification environments. Manual environment construction scales poorly.
Current approaches use string-matching verification (grep-based) instead of actually executing code, enabling agents to cheat without solving real problems.
SWE-Universe uses an autonomous agent (custom Qwen model) to build executable verification scripts from GitHub pull requests. The agent iteratively tests scripts against buggy and fixed states, with in-loop detection of superficial verification patterns.
This creates the largest real-world SWE task collection with genuine executable verification.
Extract test and fix patches from pull requests.
def process_github_pr(pr_data):
"""Extract test and fix components from PR."""
# Parse PR title and description
title = pr_data['title']
description = pr_data['body']
# Separate modified files into fix components
fix_patch = {}
test_files = []
for file_change in pr_data['files']:
if 'test' in file_change['filename'].lower():
test_files.append(file_change)
else:
fix_patch[file_change['filename']] = file_change[]
language = infer_language(pr_data[])
{
: title,
: description,
: language,
: fix_patch,
: test_files,
: pr_data[]
}
():
file_extensions = count_extensions_in_repo(repo)
primary_ext = (file_extensions, key=file_extensions.get)
extension_to_language(primary_ext)
LLM-based agent generates executable test scripts.
def autonomous_build_agent(pr_processed, building_agent, max_iterations=5):
"""Build verification script via iterative agent loop."""
context = f"""Generate a bash script that verifies this PR:
Title: {pr_processed['title']}
Description: {pr_processed['description']}
Requirements:
1. Script MUST execute actual code, not string matching
2. Test against both buggy (original) and fixed (patched) repository states
3. Output 'PASS' if fix resolves issue, 'FAIL' otherwise
"""
verification_script = building_agent.generate(context)
# Iteratively test and refine
for iteration in range(max_iterations):
# Clone repo and test script
buggy_result = run_script_on_repo(verification_script, pr_processed['repo_url'], buggy=True)
fixed_result = run_script_on_repo(verification_script, pr_processed['repo_url'], buggy=False)
# Check for hack patterns (superficial verification)
is_hacking = detect_hack_patterns(verification_script, buggy_result, fixed_result)
if is_hacking:
# Force genuine execution
feedback = f"Script uses superficial verification. Buggy: {buggy_result}, Fixed: {fixed_result}. Must execute actual code."
verification_script = building_agent.refine(verification_script, feedback)
elif buggy_result != 'PASS' and fixed_result == 'PASS':
# Converged to valid verification
return verification_script, True
return verification_script, False
def detect_hack_patterns(script, buggy_result, fixed_result):
"""Identify superficial verification (string matching vs execution)."""
# Patterns indicating string matching
hack_patterns = [
'grep -q',
'grep -c',
'echo.*|.*grep',
'if.*grep',
'test -f.*grep'
]
for pattern in hack_patterns:
if re.search(pattern, script):
# Check if results differ despite hack
if buggy_result == fixed_result:
return True # Hack didn't differentiate states
return False
Ensure scripts differentiate buggy from fixed states.
def validate_verification_script(script, repo_url, pr_data, language):
"""Confirm script properly verifies the PR fix."""
# Test on buggy version
buggy_repo = clone_repo_at_state(repo_url, 'buggy')
apply_patches_to_repo(buggy_repo, pr_data['test_files'])
buggy_output = execute_script(script, buggy_repo, language)
success_on_buggy = (buggy_output.returncode != 0) or ('FAIL' in buggy_output.stdout)
# Test on fixed version
fixed_repo = clone_repo_at_state(repo_url, 'fixed')
apply_patches_to_repo(fixed_repo, pr_data['fix_patch'])
apply_patches_to_repo(fixed_repo, pr_data['test_files'])
fixed_output = execute_script(script, fixed_repo, language)
success_on_fixed = (fixed_output.returncode == 0) and ('PASS' in fixed_output.stdout)
return success_on_buggy and success_on_fixed
Generate environments for diverse repositories and languages.
def generate_swe_universe(pr_list, building_agent, output_dir, batch_size=100):
"""Generate SWE environments for PR collection."""
successful_envs = []
for batch_start in range(0, len(pr_list), batch_size):
batch = pr_list[batch_start:batch_start + batch_size]
for pr in batch:
# Process PR
pr_proc = process_github_pr(pr)
# Build and validate script
script, success = autonomous_build_agent(pr_proc, building_agent)
if success:
is_valid = validate_verification_script(
script, pr_proc['repo_url'], pr_proc, pr_proc['language']
)
if is_valid:
# Save environment
env_id = f"{pr['repo_name']}__{pr['number']}"
save_environment(env_id, script, pr_proc, output_dir)
successful_envs.append(env_id)
return successful_envs
| Parameter | Value | Notes |
|---|---|---|
| Building agent iterations | 3-5 | Balance quality with latency |
| Hack detection patterns | Language-specific | Adapt for target languages |
| Success rate threshold | 80%+ | Before/after differentiation |
| Supported languages | Python, JS, Java, C++, etc. | Expand as needed |
| Batch size | 50-200 | Process efficiency |
SWE-Universe: Scale Real-World Verifiable Environments to Millions https://arxiv.org/abs/2602.02361