| name | eval-gated-bulk-edits |
| description | Methodology for safely running bulk LLM-driven content edits across many files using mechanical + LLM-judge gating, canary regression injection, and pilot-then-scale phasing. Trigger keywords: bulk optimization, bulk edit, eval gating, batch refactor, automated content edits, optimizer regression, pilot then scale, canary regression, cross-model judge, combined gating. |
| license | MIT |
| metadata | {"version":"1.3.0","hermes":{"tags":["Quality","Bulk-operations","LLM-evaluation","Safety"],"related_skills":["document-baseline-preservation","safe-file-deduplication"]}} |
Eval-Gated Bulk Edits
Use when applying LLM-driven edits to >10 files where mistakes are non-obvious. Skip for small batches (<10), pure mechanical transforms (formatters, linters), or changes verifiable with tests.
The four ingredients
1. Pilot-then-scale phasing
Optimize a small mixed-risk sample (8-15 files) first. Run the full eval against the pilot output. Calibrate the optimizer prompt and the eval based on what surfaces. Only then sweep the rest in batches.
The pilot's job is to validate the eval, not the optimizer.
2. Two-tier eval
- Tier 0 — mechanical preservation (this IS the
check.sh referenced in the gating table): a script that diffs old vs new and asserts invariants (every code block preserved verbatim, no semantic keyword loss, and — for file types that have them — numeric thresholds present, version bumped; skip the field-specific invariants on files that don't carry those fields rather than false-failing). Cheap, 100% coverage. Known false positives.
- Tier 1 — LLM-as-judge: prompt a fresh model with old + new + adversarial framing ("list specifically what was lost; do not say 'looks good'"). Request structured JSON output (the examples use a CLI's
--json-schema/--bare flags with ANTHROPIC_API_KEY; substitute your harness's structured-output and API mechanism — the pattern is model-agnostic, the flags are not).
3. Cross-model judge
Optimizer on Opus → judge with Sonnet (or reverse). Same-model judging compounds sycophancy. Different model + adversarial prompt breaks the loop.
4. Canary regression injection
Validate the eval, not just the optimizer. Inject 3 deliberate regressions (drop a keyword, delete a code block, paraphrase a verbatim command) into separate copies. If both eval tiers don't catch them, the eval is broken — fix it before trusting it.
Combined gating decision
check.sh | Tier 0 | Tier 1 | Action |
|---|
| FAIL | * | * | Revert (syntax broken) |
| PASS | PASS | PASS | Auto-commit |
| PASS | PASS | FAIL | Human review (judge false-positives on legitimate cleanups) |
| PASS | FAIL | PASS | Human review (Tier 0 noise) |
| PASS | FAIL | FAIL | Revert (cross-validated regression) |
Never auto-revert on Tier 1 alone. Cross-validation is what makes the gate trustworthy.
Workflow per batch
- Once, before the first batch: inject the 3 canaries (ingredient 4) and confirm both tiers catch all 3 — that gate must pass before any real sweep. Re-run it if you materially recalibrate the optimizer or eval prompt mid-sweep.
BASELINE_REF=$(git rev-parse HEAD) — the revert target: git reset --hard $BASELINE_REF (or revert the specific batch commit) on a cross-validated regression.- Spawn one optimizer subagent (your harness's agent mechanism) with the batch's file list and explicit "NEVER cut" rules (never delete a quality-floor rule, a verbatim code block, or a numeric threshold to simplify).
- Run Tier 0 + Tier 1 (parallel).
- Apply combined gating: revert failures, manually fix human-review cases.
- Commit batch. Hold the push until this batch's sanity-check clears if later batches depend on it — a pushed regression in batch N breaks dependent batches N+1.. before the every-5 review catches it; for independent batches, push freely.
- Sanity-check by hand every 5 batches.
Mechanical transforms — what "skip the gating" still requires
Pure mechanical transforms skip the LLM judge, not verification. From a 658-link bulk rewrite across 237 files:
- Dry-run first: build the full rewrite map, resolve every target (existence check), print counts + unresolved list BEFORE touching files
- Leave unresolved targets untouched — never guess; map them in a second pass with content verification (grep the target's distinctive content), and annotate genuinely-gone targets with dated notes instead of links
- Form-aware rules — handle backtick-wrapped, markdown-link, and bare occurrences separately so you never inject formatting inside code fences
- Invariants to assert after: zero resolvable patterns remain; insertions == deletions for pure in-place substitutions; fence balance unchanged on every touched file; any pre-existing defect found (e.g. an unbalanced fence at HEAD) is reported, not silently fixed or blamed on the transform
Costs
LLM judge ~$0.13/call on Sonnet with project CLAUDE.md loaded. Budget ~$0.13 × file count + pilot overhead. For 150 files, ~$20.
If cost matters: --bare mode with ANTHROPIC_API_KEY halves cost; or fall back to Haiku after pilot validates the canaries are still caught.
