setup-nightly-eval

name	setup-nightly-eval
description	Use this skill when the user wants to schedule the harness eval to run every night, asks "how do I set up nightly evals", "schedule the eval", "run evals on a cron", or "nightly regression for the harness". The kit already ships a GitHub Actions workflow at .github/workflows/eval-nightly.yml — this skill walks the user through enabling it (secret setup, smoke run via workflow_dispatch, verifying the first scheduled run). Do NOT use this skill to "remind me to run eval every night in this Claude session" — that is the /loop skill or CronCreate (which is session-only), a different concern.
allowed-tools	Read, Bash(gh workflow run:*), Bash(gh run watch:*), Bash(gh secret list:*), Bash(ls:*), Bash(cat:*), Bash(test:*)
suggested-turns	4

Background — why GitHub Actions, not CronCreate

A common request is "use CronCreate to run the eval every night". That does not do what the user wants:

• CronCreate jobs live only in the current Claude Code session. Closing the REPL deletes them. Auto-expire after 7 days regardless.
• Jobs only fire while the REPL is idle, not when the laptop is asleep or off.
• They run Claude turns, which spend tokens for every fire.

For a real nightly cadence ("runs at 6am whether I'm at the keyboard or not"), the right substrate is OS-level cron / launchd / GitHub Actions. This kit ships a GitHub Actions workflow as the default because:

1. No local daemon to babysit.
2. Free for public repos and within the free tier for most private ones.
3. Results land in workflow artifacts — visible from anywhere.

When to use

Trigger phrases (English / Vietnamese):

• "set up nightly eval" / "lập lịch eval mỗi đêm"
• "schedule the harness eval"
• "make the eval run on a cron"
• "nightly regression for the harness"

Do NOT invoke for:

• One-off ad-hoc eval runs — use /eval-runner directly.
• In-session polling ("re-run every 10 min until I say stop") — that's the /loop skill.
• Local-machine cron setup (launchd / crontab) — that path is on the user's machine and a skill cannot install OS daemons. Print the recipe and let them paste it.

Steps

1. Verify the workflow file exists. It ships via installCi: true.

   test -f .github/workflows/eval-nightly.yml && echo OK || echo MISSING
   

   If MISSING: the user opted out of CI files at scaffold time. Tell them to re-run agent-harness-kit init --yes without --no-ci (or to manually copy from node_modules/agent-harness-kit/src/templates/_ci/). The workflow should run harness readiness and git diff --check before eval work, and should conditionally install Node/Python dependencies.

2. Check the eval transport. The workflow defaults to mock transport unless ANTHROPIC_API_KEY is set in repo secrets. Decide with the user:

   • Mock (free): smoke-tests the eval runner shape — catches a broken JSONL writer, but does not exercise the model. Good default for forks / OSS.
   • claude-cli (real, costs tokens): runs the actual model on each task. Catches regressions caused by prompt/skill changes. Costs ~$0.05–0.50/night depending on task set size.

3. (If real transport) ensure the secret is set:

   gh secret list | grep ANTHROPIC_API_KEY
   

   If absent, ask the user to set it via:

   gh secret set ANTHROPIC_API_KEY
   # paste the key when prompted (it never appears in shell history)
   

4. Trigger a first manual run via workflow_dispatch so the user does not wait 24h to confirm the wiring:

   gh workflow run eval-nightly.yml --field set=quick --field transport=mock
   # then watch:
   gh run watch
   

5. Print the contract. What the user just enabled:

   ### Nightly eval enabled
   **Workflow:** .github/workflows/eval-nightly.yml
   **Cron:** 0 6 * * * UTC (offset: see `gh run list` for actual fire times)
   **Transport:** mock | claude-cli
   **Set:** quick (3 tasks) | full (all tasks)
   **Results:** uploaded as `eval-results` artifact on each run
   

Output contract

The skill prints a single block matching the shape above. Do not edit the workflow file from here — if the user wants to change the cron, the task set, or the transport default, they edit .github/workflows/eval-nightly.yml directly (it is a normal yml file, not a templated artifact, after install).

When the workflow file is owned by the kit

Re-running agent-harness-kit upgrade will refresh .github/workflows/eval-nightly.yml. If the user has hand-tuned cron or transport defaults, mention this — they should either:

• Move their customisation into .harness/config.json#evals (kit reads it on next render) and let the workflow stay vanilla, or
• Document the customisation in a comment so the next upgrade does not silently overwrite it.