| name | issue-to-eval |
| description | Converts one or more GitHub Issues into standardized benchmark data using automated scripts. Use when a user provides an issue number or URL and wants to add it to the evaluation suite. |
Issue to Eval
Converts GitHub Issues into standardized benchmark evaluation cases (JSON) and saves them to the _automation/evals/ directory. It automatically discovers new issues labeled benchmark and updates existing evaluations if the issue content has been modified.
Task Flow
Option A: Sync All Benchmarks (Recommended)
Use this to automatically identify and update all evaluations from GitHub:
python3 _automation/issue-to-eval/scripts/sync_benchmarks.py
- Discovers all issues with the
benchmark label.
- Compares each issue against the local files in
evals/.
- Adds new cases and updates existing ones if the prompt or assertions have changed.
Option B: Import Specific Issue
Use this for a one-off import or testing:
python3 _automation/issue-to-eval/scripts/import_issue_eval.py --issue {ISSUE_NUMBER_OR_URL}
Review Output
- Report the status for each issue (Success/Updated/Skipped/Error) to the user.
- If an issue is modified on GitHub, running the sync script will propagate those changes to the local evaluation suite.
Requirements
- The issue MUST follow the standard benchmark template with headers:
## Skills, optional ## Language, ## Query, ## Expected Output, ## Attached Files / Input Context, and ## Rubric Criteria (Assertions).
## Skills may contain one or more skill names, one per line or comma-separated. The generated eval stores them in target_skills.## Language is optional. When present, the generated eval stores it in language so the benchmark runner can pass the same language constraint to both agents.
