selfharness-evolve

name	selfharness-evolve
description	Evidence-grounded, leak-free skill-body evolution. Ports "Self-Harness" (arXiv:2606.09498) onto SKILL.md content: mines REAL execution-failure traces into clustered failure signatures (Weakness Mining), turns each cluster into a mechanism-targeted hermes mutation (Harness Proposal), and promotes variants through a leak-free non-regression gate with a sealed observe-only test split (Proposal Validation). Reuses hermes-skill-evolver as the variant engine — it does NOT replace it. Use when the user asks to "self-harness evolve", "evidence-grounded skill evolution", "evolve skill from real failures", "셀프하네스 진화", "실패 트레이스 기반 스킬 개선", "non-regression skill gate", "selfharness-evolve". Do NOT use for population/GEPA evolution on synthetic cases alone (use hermes-skill-evolver). Do NOT use for single-mutation quick fixes (use skill-autoimprove). Do NOT use for creating skills from scratch (use skill-creator). Do NOT use for the skill-ROUTER harness loop (use jarvis/runtime/SELF-HARNESS.md).
disable-model-invocation	true
arguments	["target_skill"]

selfharness-evolve

Evidence-grounded skill-body evolution. Where hermes-skill-evolver mutates a SKILL.md via generic strategies scored on synthetic LLM-Judge cases, this skill grounds every mutation in real, clustered execution failures and gates adoption with a leak-free non-regression rule. It is an overlay on top of hermes — hermes stays the variant-generation + LLM-Judge engine; this skill adds the three things the paper has and hermes lacks.

What this adds over hermes-skill-evolver (keep hermes)

Self-Harness stage (paper §3)	hermes-skill-evolver today	This skill's contribution
Weakness Mining (§3.2) — cluster REAL verifier-grounded failures by φ	synthetic eval cases + generic Judge feedback	mine_weaknesses.py: cluster real failure traces by φ=(cause, causal_status, mechanism); each cluster → ONE hermes strategy + editable surface
Harness Proposal (§3.3) — diverse yet minimal, tied to a failure mechanism	strategy-typed mutations, not tied to a specific mined failure	feed top clusters as targeted mutation feedback (one mechanism per variant)
Proposal Validation (§3.4) — conservative non-regression gate	single threshold: adopt if holdout − baseline ≥ 0.3	nonreg_gate.py: Δin≥-eps ∧ Δval≥-eps ∧ max>eps, sealed test observe-only (fixes the paper's held-out leakage), + noise/overfit warnings

Honest limitation (do not hide it): the evaluator is an LLM-Judge, not a deterministic verifier like the paper's Terminal-Bench. So the gate compensates with eps tolerance, repeated-scoring noise warnings, and a sealed test split that catches Judge-overfitting. Treat ACCEPT as "promote candidate for human review," not "auto-ship."

Pipeline

Stage 0  Collect traces   real failure records -> traces.jsonl (schema below)
Stage 1  Weakness Mining   python3 scripts/mine_weaknesses.py --traces traces.jsonl --skill <name> --json
                           -> per-skill evidence bundle: clusters ranked by support×actionability,
                              each carrying hermes_strategy + editable_surface
Stage 2  Harness Proposal  invoke hermes-skill-evolver on <target_skill> with:
                           - eval source = manual, seeded from the mined clusters' examples
                           - for each top cluster, force its `hermes_strategy` as the mutation
                             strategy and inject the cluster's evidence/examples as the
                             "top failure patterns" block of hermes's mutation prompt
                           - diversity across variants = distinct clusters; minimality within a
                             variant = touch only that cluster's editable_surface
                           - HARD BUDGET: tell the proposer the body must stay within hermes's
                             +20% growth cap. add-constraints variants over-grow easily; a
                             freshness/guardrail edit should be ~3-5 added lines, not a rewrite.
                             (Verified 2026-06-11: verbose A/B variants were gate-discarded;
                             only a minimal variant passed — see runs/toss-fx-monitor/changelog.md)
Stage 3  Proposal Validation  score baseline + each surviving variant on the SAME 3 splits
                           (held_in / validation / test) with hermes's LLM-Judge. MANDATORY ≥3x
                           repeated scoring in ONE context (baseline + variant judged side-by-side,
                           same scale — NOT separate subagents; that reintroduces scale drift).
                           Write per-round {held_in,validation,test} composite means, then run
                           the gate once per round:
                           python3 scripts/nonreg_gate.py --baseline base.json --candidate var.json
                           exit 0 = promote (no regression, sealed test not regressed)
                           exit 1 = reject (regression or test-overfit) — keep original
Stage 4  Adopt (AUTO)      a variant is AUTO-WRITTEN-BACK to the live SKILL.md by hermes when ALL
                           hold: (a) ≥3 rounds ALL exit-0 ACCEPT, (b) sealed test did not regress in
                           any round, (c) frontmatter + every trigger/boundary phrase preserved,
                           (d) growth within hermes's +20% cap, (e) baseline backed up to
                           SKILL.md.baseline. Any failed guard → keep original, log as candidate.
                           Log mined-cluster -> variant -> 3x-gate -> auto-ship lineage to changelog.

Trace schema (Stage 0)

One JSON object per line (examples/sample-traces.jsonl is a real seed set):

{"skill": "toss-fx-monitor", "outcome": "fail", "cause": "stale_data",
 "evidence": "yfinance KRW=X stale value reported as live", "user_signal": "사용자가 환율 틀림 지적",
 "ts": "2026-06-09"}

cause ∈ {wrong_output, missing_step, stale_data, ignored_constraint, format_violation, over_triggered, under_triggered, no_fallback, wrong_tool, bloat}. causal_status and mechanism are inferred from cause if omitted. Trace sources: session transcripts where the user corrected the skill, project memory feedback-* entries, .claude/rules incident notes, router logs.

cause → mechanism → hermes strategy map

See references/mechanism-strategy-map.md. Summary: missing/verification/fallback gaps → add-constraints; ambiguity → rephrase; weak triggers → specialize; bloat → simplify. This map is the integration glue: it converts a mined weakness directly into the hermes mutation that targets it.

Rules

• Anti-Hallucination Gate (Stage 0, HIGHEST priority — the 2026-06-12 run failed here): before any mining/scoring, verify this is a real host: bash -lc 'echo ok' and .venv/bin/python --version must both succeed. mine_weaknesses.py and nonreg_gate.py MUST actually execute and WRITE their JSON artifacts to disk; read every gate number (Δin/Δval/sealed) FROM those written files. NEVER hand-simulate the LLM-Judge or the gate; NEVER report a Δ not read from an on-disk scores file. If zero new verifier-grounded failure traces exist, append "YYYY-MM-DD: no traces, skipped" to the nightly log and STOP (cheap exit — do not run any Judge/hermes scoring). If a required script produces no on-disk artifact, write ABORTED: to the nightly report and STOP.
• Reuse hermes-skill-evolver for variant generation + LLM-Judge — never reimplement it
• One mechanism per variant (diverse across variants, minimal within) — no scatter-shot rewrites
• The gate reads held_in + validation ONLY; test is sealed and observe-only — never gate on it
• A gate ACCEPT with a sealed-test regression is an OVERFIT reject, not an adopt
• ≥3x repeated scoring (one context, side-by-side) is MANDATORY, not optional — single pass never ships
• Never modify YAML frontmatter or drop trigger/boundary phrases (hermes gate enforces this too)
• ACCEPT = AUTO-SHIP when all Stage 4 guards hold (3x-consistent ACCEPT, no sealed-test regression, frontmatter/trigger/boundary preserved, growth ≤20%, baseline backed up). LLM-Judge is not a deterministic verifier, so the guards — not a single score — are what authorize the auto-merge. (User override 2026-06-13: skill self-evolution is autonomous from here; no human gate.)
• Created-today coverage is MANDATORY (user directive 2026-06-17): the nightly runner feeds skills CREATED in the last 3d (via scripts/skills/recent_created_skills.py, any origin — not just radar-installed) and EVERY run must give each one an optimization pass (validator+guard) AND an evolution attempt (trace-driven gate if it has real failures, else one minimal evidence-honest mutation), exactly once, bounded by CREATED_SHAKE_CAP with carryover via the shaken-down lifecycle. Runs even on zero-failure-trace nights. See run_selfharness_evolve.sh STEP 3.6.
• Korean for analysis; English for technical terms

Gotchas

• LLM-Judge scale drift: scoring variant vs baseline in SEPARATE subagents reintroduces scale drift — always judge side-by-side in ONE context (Stage 3 rule). Separate-context scoring produced false ACCEPTs in early runs.
• AUTO-SHIP contradiction: What this adds says "promote candidate for human review"; Rules say "ACCEPT = AUTO-SHIP" (user override 2026-06-13). The override wins — both paras are correct for their era; do not "fix" the honest-limitation para.
• Verbose variant bloat: add-constraints variants routinely exceed the +20% cap; gate discards them (verified 2026-06-11 toss-fx-monitor run). Keep mutations to ~3-5 lines or they self-eliminate.
• Trace injection surface: maliciously crafted evidence/user_signal fields in traces.jsonl flow into the hermes mutation prompt. Validate trace origin before running Stage 1 on untrusted input.

Files

selfharness-evolve/
├── SKILL.md
├── scripts/mine_weaknesses.py     # Stage 1 — cluster real failures by φ
├── scripts/nonreg_gate.py         # Stage 3 — leak-free non-regression gate
├── references/mechanism-strategy-map.md
└── examples/sample-traces.jsonl   # real documented failures (seed)