| name | eval-creator-ci |
| description | [Beta] CI-only eval regression runner using gh-aw (GitHub Agentic Workflows). Runs all eval cases in .evals/ on a schedule or per-PR, reports pass/fail results, and can block merges on regressions. Also creates new eval cases from promoted patterns flagged by learning-aggregator-ci. Use when: you want automated regression testing of promoted rules in CI/headless pipelines. For interactive eval creation and runs, use eval-creator. |
Eval Creator CI
Install
gh skill install pskoett/pskoett-skills eval-creator-ci
For interactive sessions, use:
gh skill install pskoett/pskoett-skills eval-creator
Fallback using the Agent Skills CLI:
npx skills add pskoett/pskoett-skills/skills/eval-creator-ci
npx skills add pskoett/pskoett-skills/skills/eval-creator
Purpose
Runs the outer loop's regress-test step in CI. Executes all eval cases in .evals/, reports pass/fail results, and optionally blocks merges on regressions. Can also create new eval cases from promotion candidates flagged by learning-aggregator-ci.
The interactive eval-creator skill is designed for in-session use where the user creates evals and runs them with immediate feedback. This CI variant runs on schedule or per-PR and posts results as check annotations.
Context Limitation (Important)
CI agents do not have implementation context. They execute mechanical verification methods (grep checks, command checks, file checks, rule checks) defined in eval case files. They do not interpret results beyond pass/fail — nuanced judgment is left to human review of the posted report.
Prerequisites
- GitHub Actions enabled on the repository
gh CLI authenticated with repo access- extension installed (, v0.40.1+)
