issue-to-eval

name	issue-to-eval
description	Converts one or more GitHub Issues into standardized benchmark data using automated scripts. Use when a user provides an issue number or URL and wants to add it to the evaluation suite.

Issue to Eval

Converts GitHub Issues into standardized benchmark evaluation cases (JSON) and saves them to the _automation/evals/ directory. It automatically discovers new issues labeled benchmark and updates existing evaluations if the issue content has been modified.

Task Flow

Option A: Sync All Benchmarks (Recommended)

Use this to automatically identify and update all evaluations from GitHub:

python3 _automation/issue-to-eval/scripts/sync_benchmarks.py

• Discovers all issues with the benchmark label.
• Compares each issue against the local files in evals/.
• Adds new cases and updates existing ones if the prompt or assertions have changed.

Option B: Import Specific Issue

Use this for a one-off import or testing:

python3 _automation/issue-to-eval/scripts/import_issue_eval.py --issue {ISSUE_NUMBER_OR_URL}

Review Output

• Report the status for each issue (Success/Updated/Skipped/Error) to the user.
• If an issue is modified on GitHub, running the sync script will propagate those changes to the local evaluation suite.

Requirements

• The issue MUST follow the standard benchmark template with headers: ## Skills, optional ## Language, ## Query, ## Expected Output, ## Attached Files / Input Context, and ## Rubric Criteria (Assertions).
• ## Skills may contain one or more skill names, one per line or comma-separated. The generated eval stores them in target_skills.
• ## Language is optional. When present, the generated eval stores it in language so the benchmark runner can pass the same language constraint to both agents.