| name | overnight-eval |
| description | Long-running evaluation campaign with tmux isolation. Use when running any eval batch exceeding ~30 minutes, canonical benchmark runs, or when the user will be away during execution. Pre-flight verification, tmux launch, monitoring, post-flight analysis. NOT for: short interactive runs (use eval-run instead).
|
| auto-activate | false |
Overnight Eval Campaign
Use when launching a long-running evaluation batch that needs tmux deployment
or should run unattended (overnight, weekend).
Trigger: When user types /overnight-eval with optional arguments.
Arguments
$ARGUMENTS — shorthand or explicit flags for the eval script.
If omitted, prompt interactively.--dry-run — run pre-flight only, do not launch--session-name <name> — tmux session name (default: eval-campaign)--alert-threshold <N> — error rate % that triggers alert (default: 50)
Iron Law
NO EVAL CAMPAIGN WITHOUT PRE-FLIGHT VERIFICATION.
Every API key, environment dependency, dataset, and eligibility check MUST pass
before the batch command is issued. No exceptions.
Anti-Rationalization Table
| Excuse | Reality |
|---|
| "I'll check the API keys later" | A failed key means wasted hours and zero results |
| "The environment is probably fine" | 2 seconds to verify. Hours to re-run |
| "I can run it in the foreground" | Terminal disconnect = killed process. Always tmux |
| "Resume will catch anything I miss" | Resume perpetuates pre-flight errors |
| "I'll analyze tomorrow" | Tomorrow you won't remember the parameters. Log now |
Workflow
Phase 1: Pre-Flight Verification
Run ALL checks. Every check must pass before proceeding.
- Environment: Verify venv/dependencies, Python version, key packages
- Hardware (if applicable): GPU availability, memory, utilization
- API Keys: Check only for selected models
- Dataset/Source: Verify benchmark data exists and is accessible
- Eligibility: Verify task list, apply exclusions
Display pre-flight summary and wait for user confirmation.
If --dry-run, stop after displaying summary.
Phase 2: Launch in tmux
mkdir -p <project-root>/logs/eval-staging
tmux new-session -d -s <session-name> -c <project-root>
tmux send-keys -t <session-name> "<eval-command> 2>&1 | tee logs/eval-staging/campaign_$(date +%Y%m%d_%H%M%S).log" Enter
Verify tmux session is running before proceeding. Do not write campaign logs
directly under results/; keep raw run logs in staging until post-flight review.
Phase 3: Monitor (Periodic)
The user can check progress at any time:
- Is the tmux session still alive?
- How many results completed?
- Current pass/fail rates
- Error spike detection (alert if error rate exceeds threshold)
Phase 4: Post-Flight Analysis
After completion:
- Run analysis scripts
- Review staged campaign log for anomalies
- Generate summary with results table
- Alert user (desktop notification if available)
Phase 5: Handoff
Present results summary and recommended next steps:
- Promote reviewed artifacts into
results/ only through the approved results ingestion path
- Refresh dashboard if applicable
- Commit results
- Update paper sections if relevant
