ate

name

ate

description

ATE (Automated Testing Environment): drive long-running (2-3hr+) unattended variation-test suites against the Aider, OpenRouter, and GLM 5.2 pipeline using a local Gemma model in LM Studio as the worker, with periodic frontier-model check-ins for drift control, ending in a single triaged GitHub issue. Use this whenever the user wants to "run variation tests overnight/for hours", "debug a bunch of small Aider bugs", "test Aider flag combinations", "fuzz the Aider harness", mentions LM Studio + Gemma as a local test driver, wants test failures rolled up into one GH issue triaged by severity, or invokes ATE by name. Also trigger if the user asks to set up a check-in loop between a local model and a frontier model for a long-running task.

ATE — Automated Testing Environment (generic bounded-variation matrix runner)

The skill interface lives here; its implementation is bundled under utils/ate/.

ATE walks a declared grid of command variations unattended for hours, logs every result as structured JSON, and a frontier model (you, Claude) checks in every ~5 minutes to catch drift/looping. The full cycle — run, capture, document, file — is chained end to end: when the run ends (time limit, abort, or safety cap), run_variations.py automatically hands off to compile_issue.py, which opens one GitHub issue titled ATE - [test-name] yyyy-mm-dd containing every finding from that run as a single checklist, ranked by severity (critical first). No manual second step required as long as --gh-repo was passed.

Architecture (linear)

1. Gemma (LM Studio, local) — worker. Cycles through the variation grid (repeating from the top once it reaches the end), runs aider --model openrouter/z-ai/glm-5.2 ... as a subprocess, classifies the result.
2. error_log.jsonl — the single source of truth. Append-only. Every variation writes one line here regardless of pass/fail.
3. control.json — the only channel Claude writes to. Gemma polls it before each iteration. {"action": "continue"} / {"action": "abort", "reason": "..."}. run_variations.py resets this to continue at the start of every run, so a stale abort from a previous run in the same directory can't kill a fresh one.
4. Claude (you) — supervisor. Every ~5 min, run checkin.py, read the summary, decide continue/abort, write control.json if aborting.
5. compile_issue.py — chained automatically: run_variations.py calls it the moment the run stops (time limit, abort, or safety cap), passing --gh-repo through. It groups error_log.jsonl by severity/signature into one unified, severity-ranked checklist and opens a single GitHub issue titled ATE - [test-name] yyyy-mm-dd via gh issue create. Can still be run standalone for a manual rollup (e.g. after a crash that killed run_variations.py before it could chain).

Quick start (do this first)

# 1. Install
bash utils/ate/install.sh           # copies the interface + implementation to ~/.claude/skills/

# 2. In LM Studio: load a Gemma 4 model (31B Dense recommended), start the
#    Local Server (Developer tab -> Start Server). Default: http://localhost:1234/v1
#    Note the exact model identifier shown in LM Studio's server log/model list.

# 3. Set OpenRouter key for the Aider side of the pipeline being tested
export OPENROUTER_API_KEY="sk-or-v1-..."

# 4. Point run_variations.py at a scratch git repo (never your real repo).
#    error_log.jsonl and control.json land in this directory by default —
#    run checkin.py from here too, or pass --log/--control explicitly.
cd ~/scratch/aider-test-repo && git init -q

# 5. Kick off the run (defaults to 3 hours, edit variations.yaml first).
#    --gh-repo is what turns on the automatic issue filing at the end — omit it
#    to just get error_log.jsonl with no GitHub side effect. Requires
#    `gh auth status` to already be logged in.
python3 ~/.claude/skills/ate/scripts/run_variations.py \
  --repo . \
  --variations ~/.claude/skills/ate/variations.example.yaml \
  --lmstudio-model "gemma-4-31b-instruct" \
  --gh-repo OWNER/REPO \
  --test-name "aider-flag-fuzz" \
  --minutes 180 &

Then, on your (Claude's) side, every ~5 minutes:

python3 ~/.claude/skills/ate/scripts/checkin.py --tail 20

Read the printed summary. If it looks healthy, do nothing (Gemma keeps polling control.json, sees no abort, continues). If you see drift (see "What counts as drift" below), abort the run — checkin.py --abort "reason" does this for you in one call. There is no "redirect" action; the only control signals are continue and abort.

When the run ends (time limit, abort, or the iteration safety cap of 20 full passes over the grid — see "Notes / gotchas"), run_variations.py automatically chains into compile_issue.py and opens one GitHub issue titled ATE - [aider-flag-fuzz] 2026-07-09 (using today's date and the --test-name you passed, or the variations.yaml filename's stem if you didn't pass one) with every finding from the run as a single severity-ranked checklist. Nothing further to run by hand. If you need the standalone/manual form (e.g. run_variations.py itself crashed before it could chain, or --gh-repo was omitted):

python3 ~/.claude/skills/ate/scripts/compile_issue.py \
  --log error_log.jsonl --repo OWNER/REPO --test-name "aider-flag-fuzz"

Exit codes (#142)

Both scripts in the filing chain exit with a contract a wrapper, CI job, or supervising agent can branch on. Before #142 every terminal state exited 0 — including a failed gh issue create — so a multi-hour run's final step could fail invisibly.

Code	Meaning
0	Issue filed, or --dry-run body rendered.
3	No records in the log — nothing to file. Distinct from filed so callers can tell "done its job" from "did nothing".
1	gh issue create failed. issue_body.md is preserved in the working directory for manual filing.
2	run_variations.py only: repo guard refused the run (--repo is the harness itself, or carries a remote without --allow-destructive-reset).

run_variations.py propagates compile_issue.py's code when --gh-repo is set: the run's exit code IS the filing outcome. Supervisors: treat 0 and 3 as healthy ends, 1/2 as needing attention, and abort-drift separately via checkin.py. The gh issue create call is capped at 120s (ATE_GH_TIMEOUT_S overrides); a hang, a missing gh, or a launch failure all land in exit 1 with issue_body.md preserved — no traceback, no indefinite hold on the run's completion.

Targets beyond Aider (#141 Phase 5)

The engine is target-agnostic: any grid with a command_template (argv-list form — values are substituted per-token, never through a shell string) runs against any CLI. Aider is one preset, not the subject.

• Classifier oracle decoupling (expects_edits) — the stock Aider grid expects the tree to change, so exit-0-with-no-edit classifies fail/no_edit. A grid that only probes (usage surfaces, read-only diagnostics) declares expects_edits: false, and exit-0-no-edit is a PASS. The #146 Gemma soak recorded 17 false HIGH no_edit verdicts before this key existed — do not run a non-edit grid without it.
• Rollup labels are neutral by default — bug, nothing else. The Aider preset opts back in via issue_labels: [bug, aider-pipeline] in its grid; any grid or --issue-label overrides. A turn-shim soak must never file Aider-labelled issues.
• Declared non-Aider grids shipped here: variations.tool-density.yaml / variations.tool-calling.yaml (script_runner benchmarks) and variations.turn-shims.yaml (turn-shim CLI-contract usage soak — read its header for the safe execution profile before wiring it to a runner; it is diagnostic, expects_edits: false, and must run against stubs or a disposable clone if extended past the usage surface).

What counts as drift (abort triggers)

• 3+ consecutive iterations with the same likely_cause signature — Gemma is stuck re-testing the same bug instead of moving to the next variation
• Gemma marks something pass but the raw Aider stdout/stderr in the same record contains a traceback or non-zero exit code — classification is unreliable, stop and inspect
• No new lines appended to error_log.jsonl for 2+ check-in cycles — the worker has hung (check the LM Studio server and the aider subprocess). checkin.py has no memory of previous invocations, so use its printed log_last_modified_seconds_ago field and compare it against what you saw last check-in rather than relying on the tool to flag this automatically.
• Wall-clock time on a single variation exceeds --per-variation-timeout (default 180s) — treat as a hang, not a slow test; run_variations.py kills it and logs it as severity: high, category: timeout automatically

variations.yaml format

Grid of Aider flags to combine (see variations.example.yaml). Each combination becomes one subprocess call. Keep the task message short and deterministic (a fixed small task like "add a docstring to foo()") so failures are attributable to the pipeline (Aider/OpenRouter/GLM plumbing), not to task ambiguity.

model: openrouter/z-ai/glm-5.2
edit_formats: [diff, whole, udiff]
map_tokens: [0, 1024, 4096]
auto_commits: [true, false]
message: "Add a one-line docstring to the function `foo` in sample.py"
per_variation_timeout_seconds: 180

Severity rubric (used by compile_issue.py to rank the unified checklist)

Severity	Definition
critical	Aider process crashes, non-zero exit with traceback, or corrupts the git working tree
high	Wrong/no edit applied, OpenRouter auth or routing failure, timeout/hang
medium	Edit applied but malformed (bad diff format, partial file write)
low	Cosmetic — extra output noise, formatting drift, slow-but-succeeded

Files in this skill

• scripts/run_variations.py — the Gemma-driven worker loop (long-running); chains into compile_issue.py automatically when it stops, if --gh-repo was passed
• scripts/checkin.py — Claude-side supervisor: summarize + optionally abort
• scripts/compile_issue.py — rolls up error_log.jsonl into one GH issue titled ATE - [test-name] yyyy-mm-dd; runs standalone too, for a manual rollup
• variations.example.yaml — starter grid, copy and edit per pipeline under test
• utils/ate/install.sh — copies the interface and implementation to ~/.claude/skills/

Notes / gotchas

• LM Studio's OpenAI-compatible endpoint ignores auth — no API key needed locally, but you must pass the exact model name LM Studio reports (/v1/models will list it).
• Aider's OpenRouter model string needs the openrouter/ prefix on top of the OpenRouter slug itself: openrouter/z-ai/glm-5.2 — the bare slug alone will fail.
• Gemma 4's training cutoff is Jan 2025 — if Aider's CLI flags have changed since, Gemma may generate stale flag names. run_variations.py only ever calls flags from variations.yaml (never lets Gemma invent flags), which sidesteps this.
• Run everything against a disposable scratch repo. run_variations.py does a destructive git reset --hard + git clean -fdx before every variation (so each variation starts from the same pristine state and results are comparable across the grid). This permanently deletes uncommitted and ignored files in --repo. Two guards refuse the obvious footguns before anything runs (GH-195): it hard-refuses --repo pointed at the harness clone itself, and refuses a --repo that has a git remote (i.e. looks like a real checkout) unless you pass --allow-destructive-reset. A fresh git init scratch repo has no remote, so the Quick start above needs no extra flag — and a git init with no commits yet no longer crashes (the script auto-creates a base commit to rewind to).
• The grid is cycled, not walked once — run_variations.py loops back to the start after the last combination and only stops on the time budget, an abort, or an iteration safety cap of len(combos) * 20 (a backstop against a runaway --minutes value).