claude-optimizer-skill

name

claude-optimizer-skill

description

Measurement-driven optimization loop for a specific function, algorithm, or pipeline. Given a target piece of code, generates a correctness test and a benchmark, presents both for user review listing exactly what is checked and how, and after approval iterates — one change at a time, keeping only improvements above the observed noise floor that reproduce on a second benchmark invocation and preserve correctness against an immutable oracle, logging every experiment including discards. Supports two modes — interactive co-drive, or autonomous long-running (8–12h default) with an upfront session charter, silent loop, web-search-driven hypothesis generation, and hard-blocker-only escalation. Trigger when the user asks to optimize, speed up, reduce memory of, or benchmark a specific piece of code, requests an overnight or long unattended optimization run, or invokes /claude-optimizer-skill.

claude-optimizer-skill

Iterative measurement-driven optimization. The metric is the judge. Every change is kept only if it improves the metric above the measured noise floor, is reproduced on a second benchmark invocation, and does not break correctness against an immutable oracle. Discards are as valuable as wins and are logged with reasons.

Workflow

1. Mode pick — ask the user: interactive (co-drive) or autonomous (long unattended grind).
2. Setup — analyze target code; draft correctness test + benchmark; present for user approval listing what is checked and how. In autonomous mode, the approval also covers the full-session charter (time/stall budgets, permissions, escalation rules).
3. Baseline — after approval, run the benchmark on unmodified code, record the starting metric.
4. Loop — propose one change, implement, run correctness, run benchmark, keep/discard, append to log.
5. Stop — stall / time budget exhausted, correctness regression, known lower bound, or (interactive) user says stop.

---

Operating modes

Two modes. Ask the user which before touching code, in one short question:

Interactive or autonomous?

• Interactive (default) — one-line summary after each iteration; you can redirect at any point. Best for short sessions and co-driving.
• Autonomous — you approve one session charter up-front; I then grind for a fixed wall-clock budget (default 8–12h), generating hypotheses, searching the web for techniques, iterating silently, and only pinging you on hard blockers. Best for overnight / unattended runs.

Default to interactive if unspecified. Mode is fixed for the session — switching requires a new charter.

Autonomous mode — session charter

The approval step (§1.4) is extended into a single upfront charter that grants the loop everything it needs to run unattended. After approval, do not ask the user anything outside the hard-blocker list below.

The charter, shown alongside the correctness + benchmark plan, must include:

• Time budget — wall-clock hours (default 10h; user overrides).
• Stall budget — consecutive discards before auto-brainstorming structural changes (default 8) and before giving up (default 20).
• Permissions, pre-granted for the whole session:
  • Write files within the project: oracle/, iters/, the harness files, the log, environment.txt.
  • Run the correctness and benchmark harnesses, and the project's build system, as many times as needed.
  • Git operations on a dedicated branch coptimize/<target>-<YYYYMMDD> — one commit per keep; no push, no rebase of shared history.
  • WebSearch + WebFetch for optimization techniques, reference implementations, published benchmarks, compiler intrinsics, SIMD/vector patterns, and academic papers relevant to the target.
• Explicitly out-of-session (will trigger the hard-blocker list): installing system packages, modifying global config, touching files outside the project tree, force-pushing, deleting untracked files, sudo, interactive debuggers.
• Hard-blocker escalation list (ONLY these stop the loop and ping the user):
  1. Harness sanity-check fails (§1.5) or null-experiment fails (§1.6) — session cannot start.
  2. Project build breaks and cannot be restored from git within 3 attempts.
  3. Correctness regression detected in a previously-kept candidate (seed revalidation or re-run) — possible silent breakage.
  4. Time budget exhausted.
  5. Stall budget exhausted (default 20 consecutive discards with no structural idea landing).
  6. An action the charter does not cover is needed.
• Reporting cadence:
  • Silent during the loop — no per-iteration chatter.
  • Progress digest appended to autoresearch-log.md every 10 iterations or every 1h wall-clock, whichever first. User can read the log at any time without interrupting.
  • Final report on session end: baseline → final metric, total iterations, keep/discard ratio, top 5 wins, top 3 dead ends with reasons, unexplored ideas, recommended next session.

Before accepting the charter, Claude prints the full list above with the user-supplied values filled in and waits for one explicit approve / modify / cancel. Nothing is created until approval.

Practical note: Claude Code's own permission prompts operate independently of this charter. For a genuinely uninterrupted run, ask the user to configure /permissions (or .claude/settings.json) to allow the relevant Bash, Edit, Write, WebSearch, WebFetch, and git commands before approving the charter.

Autonomous mode — hypothesis generation

In interactive mode the user can redirect (try X next). In autonomous mode Claude must generate hypotheses itself. Use, in order:

1. Profile the current best. Re-profile every 5 keeps — hot paths shift after each win.
2. Code inspection. Branch misprediction hints, allocations in the hot path, redundant work, cache locality, data layout, false sharing.
3. Web search when local ideas run dry or the target enters a known-hard domain. Concrete queries: "<algorithm> SIMD implementation", "<language> micro-optimization <operation>", "fastest <problem> benchmark", authors worth grepping (Agner Fog, Lemire, Bernstein, Gueron). Record each search + the link taken forward in the log's Research column.
4. Reference implementations. If a published benchmark shows a library is clearly faster, read its source, extract the technique (license-aware — paraphrase, don't copy), translate to the target.
5. Auto-brainstorm on stall. At the auto-brainstorm threshold (default 8 consecutive discards), write a 5-bullet "what I've tried and why it didn't work" note to the log, then pivot to a structural change class not yet attempted.

---

Phase 1: Setup

1.1 Understand the target

Read the code the user pointed at. Identify:

• Language / runtime — determines benchmark harness (picks template from templates/).
• Surface — function signature, what's called, inputs/outputs, pure vs stateful.
• Plausible metric — latency per call? throughput? peak memory? p99? binary size? Pick one scalar.
• Plausible input distribution — uniform random, Zipfian, bimodal, production-shaped, etc.
• Oracle — the "known good" implementation the correctness check compares candidates against. Usually the current target code, snapshotted at session start. Sometimes a trusted library or a specification-derived implementation. Immutable for the duration of the session.

If any of these is unclear, ask one batched round of clarifying questions before writing anything. Never guess the metric.

1.2 Draft correctness test

Follow correctness-rules.md. Minimum requirements:

• Deterministic PRNG with fixed seed.
• Thousands of random inputs matching the realistic distribution.
• Plus a hand-crafted edge-case set (zero, boundaries, powers of two, min/max, NaN, empty, known past-bug cases).
• Compare candidate vs oracle via a clearly-declared mode (bitwise equality / tolerance ε / order-agnostic / structural).
• Fail hard: print first mismatch (input, expected, got), exit non-zero.

1.3 Draft benchmark

Follow benchmark-rules.md. Minimum requirements:

• Same deterministic input generation as correctness (or separate fixed-seed set of same distribution).
• Warmup 3–5 iterations (more for JIT languages).
• 20+ measurement runs. Report median (primary), min, stddev. Never mean.
• Stddev must be <5% of median; warn if not.
• Total runtime under 30 seconds.
• Oracle and candidate measured in the same process.

Selecting the harness template:

1. Check templates/ for a dedicated file for the target language/runtime:
   • c.c — C / C++
   • python.py — Python
   • java-jmh.java — JVM (Java / Kotlin / Scala)
   • csharp.cs — .NET (C# / F# / VB)
2. If present: adapt that template, keeping its shape. Record template: <name> in the approval step.
3. If absent: open templates/other-languages.md and follow Steps 1–5. The approval step MUST include the filled Concern → Answer table from Step 2 and the Harness template used: custom (no template for <language>) line from Step 5.

Do not write a harness from scratch without either a template or the filled table. Missing-template is not license to skip the analysis — it is an instruction to do it explicitly and show your work.

1.4 Present for approval (this step is mandatory)

Before creating any files, show the user this exact structure, filled in:

## Proposed setup

### Correctness test — what it verifies
- [invariant 1, e.g. output matches oracle exactly for all inputs]
- [invariant 2, e.g. function is idempotent]
- [edge cases covered: list them]

### Correctness test — how it verifies
- Input generation: [PRNG name + seed], N=[number] random inputs
- Plus hand-crafted edge cases: [enumerate them]
- Comparison mode: [bitwise equality | tolerance ε=... | order-agnostic | structural]
- Failure mode: [abort on first mismatch | collect first 10 then abort]

### Benchmark — what it measures
- Metric: [ns/op, ops/sec, bytes, MB/sec, etc.]
- Workload: N=[number] inputs, [distribution], seed=[value]

### Benchmark — how it measures
- Warmup: [K] iterations
- Measurement: [R] runs; reports median, min, stddev
- Guards: stddev <5% of median, total runtime <30s
- Harness: [template name, or "custom (no template for <language>)"]
- If custom: paste the filled `Concern → Answer` table from `other-languages.md` Step 2 here

### Files to create
- `[path/to/correctness.ext]`
- `[path/to/benchmark.ext]`
- `autoresearch-log.md`

### Out of scope
- [anything the user might expect but this setup won't catch — be honest]

**OK to proceed?** Flag anything to change: metric choice, input distribution, edge cases, tolerance, sample size, etc.

If the session mode chosen earlier is autonomous, append the full session-charter block from "Operating modes → session charter" (time budget, stall budget, permissions, escalation list, reporting cadence) after "Out of scope" and before the OK line. One approval covers everything.

Wait for explicit approval. If the user changes anything, re-present the updated plan (they review once more). Do not create files until OK.

1.5 Create harness and snapshot the oracle

After approval:

1. Snapshot the oracle. Copy the target's original source file(s) to oracle/<original-filename> inside the project. The oracle is immutable for the entire autoresearch session: it is the single source of truth for correctness and is never modified, no matter how many iterations succeed. If the user explicitly wants to reset the baseline (e.g. a structural rewrite), that is a new autoresearch session with a new oracle.
2. Create the harness. Both files from the approval plan.
   • The correctness harness imports the oracle and the candidate, and always compares candidate vs oracle. Never vs "current best" or "previous iteration".
   • The benchmark harness may report speedup vs the current best (for progress tracking), but correctness still runs against the oracle first.
3. Sanity-check: run both.
   • Correctness check → must report 0 mismatches.
   • Benchmark → must finish under the runtime cap with stddev <5%.
4. Dump environment. Create environment.txt per reproducibility.md. Log the filename in autoresearch-log.md.

If correctness or benchmark sanity-check fails, the harness is wrong (not the target). Pause the loop. Show the user what failed, propose a fix to the harness, get approval, re-run. Do not start iterating on optimizations until the harness is clean.

1.6 Null-experiment calibration

Before declaring the harness trusted and moving to Phase 2, run one null iteration:

• Candidate = byte-identical copy of the oracle.
• Expected result: speedup in [1.00x - noise_floor, 1.00x + noise_floor], i.e. within [0.98, 1.02] when stddev <5%.

If measured speedup falls outside [0.98, 1.02] with stddev <5%, the harness is noisier than it reports: there is a systematic bias (cold-cache asymmetry, ordering effects, JIT not at steady state, thermal drift) that will contaminate every subsequent result. Do not enter Phase 2. Report the deviation to the user and propose fixes: more warmup, larger N, paired/interleaved measurement, taskset/isolcpus, close background processes.

Record this calibration in autoresearch-log.md as a dedicated row labelled iter 0.5 — null-experiment with the observed speedup and stddev.

---

Phase 2: Baseline

Run the benchmark on the unmodified target. Record as iteration 0 in the log. This is the anchor for every future comparison.

---

Phase 3: The loop

3.1 One change per iteration

Non-negotiable. If two ideas seem related, do one, land it, then try the other. Bundling hides regressions — a 15% win combined with a 5% loss looks like a 10% win.

3.2 Iteration steps

1. Hypothesize — one sentence: "I'll try X because I expect Y." Announce it to the user before implementing (they can redirect; they don't need to approve).

2. Implement — modify the candidate code only. Never touch the harness.

3. Correctness — run the check. If it fails: revert immediately, log as discard — correctness broken, continue. Do not debug inside the loop.

4. Benchmark — run it. Compare median against the current best.

5. Decide. Compute the keep threshold from observed noise, not a hardcoded number.

   Inputs:

   • med_best, stddev_best — from the current best's most recent benchmark JSON (for iter 1 the current best is the oracle; thereafter it is the last-kept candidate).
   • med_cand, stddev_cand — from this iteration's benchmark run.

   Formula:

   improvement_pct = (med_best - med_cand) / med_best * 100
   noise_floor_pct = 2 * max(stddev_best / med_best, stddev_cand / med_cand) * 100
   keep_threshold  = max(2.0, noise_floor_pct)
   

   Then:

   • improvement_pct >= keep_threshold and reproduced on a second full benchmark invocation (new process, not just more runs in the same process): keep. Update the current best to this iteration's numbers.
   • improvement_pct >= keep_threshold but second invocation disagrees (result no longer above threshold): discard — flaky.
   • improvement_pct < keep_threshold: discard — within noise.
   • improvement_pct < 0 (regression vs current best): discard, revert.

   The second-invocation rule is a reproducibility gate — see benchmark-rules.md "A result counts only if". A single benchmark process can be lucky; a second cold process cannot be lucky the same way twice.

   If the provided template benchmarks candidate vs oracle (which is the default), also record vs Baseline = med_oracle / med_cand for the log. The decision above is separately about vs Best, which is what controls keep/discard.

6. Log — append a row to autoresearch-log.md including both improvement_pct and keep_threshold actually used (so a reader sees why the call went the way it did).

7. Next — propose the next experiment, stating the hypothesis.

3.3 Log format

Use log-template.md as the starting point for autoresearch-log.md. It includes the Environment section (required by reproducibility.md), an iteration table with stddev% and raw-JSON columns, and an example of how to record null-experiment (iter 0.5) and harness re-baseline rows.

Shortened preview of the table; see the template for the complete column set:

| Iter | Change                          | Metric   | stddev% | vs Baseline | vs Best | Status                                  | Raw                 |
|------|---------------------------------|----------|---------|-------------|---------|-----------------------------------------|---------------------|
| 0    | Baseline                        | 31.4 ns  | 2.1%    | 1.00x       | —       | baseline                                | iters/iter-00.json  |
| 0.5  | Null-experiment                 | 31.5 ns  | 2.3%    | 1.003x      | —       | calibration — harness trusted           | iters/iter-00.5.json|
| 1    | Branchless modular reduction    | 21.6 ns  | 1.8%    | 1.45x       | 1.45x   | keep (improvement 31.2% > floor 4.6%)   | iters/iter-01.json  |
| 2    | Left-to-right modpow            | 25.7 ns  | 2.4%    | 1.22x       | 0.84x   | discard — regression, hurts ILP         | iters/iter-02.json  |

3.4 Cadence with the user

Interactive mode: run iterations autonomously but not silently. After each iteration emit a one-line summary:

• iter N: <change> → <metric> (<status>). Next: <one-sentence hypothesis>.

The user can say continue, redirect (try X next), or stop. If 5 iterations pass with all discards, pause on your own and ask: "5 consecutive rejects. Want me to keep grinding or step back and brainstorm structural changes?"

Autonomous mode: silent during the loop. Skip the announce-hypothesis step (§3.2 step 1) and the per-iteration summary line — both go to the log instead of chat. No pause at 5 discards; at the auto-brainstorm threshold (default 8), write the brainstorm note into the log and pivot without pinging the user. Progress digest to the log per the charter cadence. Only the hard-blocker list (see "Operating modes") may interrupt.

---

Phase 4: Stop and revalidate

4.1 Seed revalidation (every 5 keeps)

After every 5th keep iteration, before proposing the next change, re-run the full benchmark with a different PRNG seed (e.g. seed=43 instead of 42). Compare the cumulative speedup under the new seed to the cumulative speedup under the original seed.

• If the two speedups agree within 15%: continue, the optimizations generalize across the input distribution.
• If the gap is larger in interactive mode: pause the loop, report to the user, and consider reverting the most seed-specific keeps.
• If the gap is larger in autonomous mode: auto-revert the single most seed-specific keep (the one whose removal closes the gap most), log the revert, and continue. Escalate to the user only if a correctness regression shows up in the revalidation run — that is hard-blocker #3.

Record the revalidation in autoresearch-log.md under Seed revalidations: regardless of outcome or mode.

4.2 Stop conditions

Stop and summarize when ANY of:

Interactive mode:

• 5 consecutive discards with no improvement.
• Metric within ~5% of a known lower bound (memory bandwidth, ALU throughput, theoretical algorithmic limit) — say so if you know one.
• Seed-revalidation (4.1) gap exceeds 15% and the user chooses not to continue.
• User says stop.
• Long session with a clear plateau (~30+ min of discards).

Autonomous mode:

• Time budget exhausted (charter default 10h).
• Stall budget exhausted (charter default 20 consecutive discards).
• Metric within ~5% of a known lower bound.
• Any entry from the hard-blocker list.
• ("User says stop" does not apply — the user is not watching. They can still abort externally by stopping the session, but the skill does not poll for it.)

Final summary should include: baseline metric, final metric, total speedup, iterations run, keep/discard ratio, top 3 wins with one-line explanation each, seed-revalidation outcomes, and a list of ideas worth trying in a future session.

---

Non-negotiable rules

1. Ask for approval of correctness test + benchmark before creating files. Listing what is verified and how.
2. One change per iteration. Never bundle.
3. Correctness before timing. Correctness failure = instant discard, no debugging.
4. Discard neutral changes. "Didn't hurt" is not a reason to keep complexity.
5. Harness mutation requires explicit re-baseline. If the harness needs to change mid-session (e.g. dead-code elimination started deleting work, input distribution no longer exercises the hot path, timer resolution too coarse, GC pressure appeared): a. Pause the loop immediately. b. Describe the harness problem and the proposed change to the user; get explicit approval. c. After the harness change, re-run the baseline against the oracle. The new measurement becomes the new anchor. d. Log the harness change as a dedicated row with status harness — re-baseline and a reason column. e. All vs Baseline ratios recorded from this point reference the new anchor. Do not retroactively recompute prior ratios — they stay as-is and remain valid under their original harness.
6. Log every iteration, including discards, with a one-line reason. The rejection log is the most valuable artifact.
7. Don't trust theory over measurement. Manual inlining, branchless code, "better" algorithms routinely lose to the compiler/JIT. Measure.
8. Mode is fixed per session. Ask the user which mode at the start. In autonomous mode, the charter is a single approval gate — do not escalate mid-session except for the hard-blocker list. Switching modes requires starting a new session with a new charter.

---

References

• correctness-rules.md — full checklist for a trustworthy correctness harness
• benchmark-rules.md — full checklist for a trustworthy benchmark
• log-template.md — log file structure
• templates/c.c — C harness skeleton
• templates/python.py — Python harness skeleton
• templates/java-jmh.java — JMH (JVM) harness skeleton
• templates/csharp.cs — BenchmarkDotNet (.NET) harness skeleton
• templates/other-languages.md — analysis checklist for languages without a dedicated template
• reproducibility.md — environment dump + platform hardening requirements