edit-manifest

name	edit-manifest
description	Implements a predict-verify loop for harness edits. Every SKILL.md, rules/*.md, or CLAUDE.md change is paired with a predicted impact stored in a manifest. The next session verifies predictions against actual outcomes, and a validation gate accepts edits only when improvement is measurable. Runs automatically on harvest-loop Step 0-c and on explicit invocation.
user-invocable	true
allowed-tools	["Read","Write","Edit","Grep","Glob","Bash"]
model	sonnet

edit-manifest — Predict-Verify Loop for Harness Edits

Implements two mechanisms from frontier harness research, named as in their source papers (both terms are the papers' own — verified 2026-06-17 against arXiv full text):

• Change Manifest (AHE, arXiv:2604.25850): each edit attaches a manifest entry declaring a falsifiable predicted impact (expected fixes + at-risk regressions); the next round intersects it with observed task-level deltas to produce a per-edit verdict. (Paper: "the Evolve Agent produces a change manifest…".)
• Validation Gate (SkillOpt, arXiv:2605.23904): a candidate is accepted only when its selection-split score is strictly greater than the current selection score (ties rejected); rejected edits are kept in an epoch-local buffer, not silently discarded. (Paper: "validation gate" / "held-out gate".)

FH's regression_guard.sh catches backward regressions. edit-manifest closes the forward loop: did the edit actually improve what it claimed to improve?

Boundary vs. verify-bidirectional: verify-bidirectional is reactive — it updates the baseline when the user raises a counter-argument to an AI recommendation. edit-manifest is predictive — it records a forward prediction at edit time and verifies it against outcomes later, with no user counter-argument required. Different axes: one responds to challenge, the other tests a self-declared hypothesis.

Manifest File Location

tracks/_meta/edit_manifest.yaml

Single append-only file. Entries are never deleted — they accumulate as the harness's edit history and negative-feedback buffer.

Manifest Entry Format

- id: em-{YYYY-MM-DD}-{slug}
  date: YYYY-MM-DD
  file: plugins/fh-meta/skills/{skill}/SKILL.md   # relative to repo root
  edit_summary: "added 'trigger phrase X' to natural language triggers"
  predicted_impact: "users entering via phrase X will increase — estimate +1 session/week"
  predicted_measurable_by: "session start logs or user utterance pattern in next 2 sessions"
  validation_status: pending   # pending | verified | falsified | untestable
  validation_type: judged      # mechanical (grep/count/git) | judged (cited observation) | untestable
  baseline_value: null         # number, when a metric exists (mechanical)
  measured_value: null         # filled at VERIFY
  delta: null                  # measured_value - baseline_value, or null
  match_score: null            # at VERIFY: 1.0 confirmed | 0.5 partial | 0.0 contradicted
  verified_at: null
  verification_note: null      # one-line MEASURED outcome + cited evidence (never bare "seems better")
  gate_decision: null   # accepted | redefine | rejected

Status vocabulary is canonical and machine-greppable — validation_status MUST be one of pending | verified | falsified | untestable, never freeform prose. The verify pass (Step V1) greps these literals; a freeform status like "predicted — verify next session" is invisible to the grep and silently never closes the loop (the format-reconciliation bug fixed 2026-06-16 — put the prose in predicted_measurable_by, keep validation_status: pending).

baseline_value / measured_value / delta apply to mechanical entries only; judged and untestable entries leave them null. match_score is the gate input (Step V3) for all types — the numeric fields are a mechanical-entry audit detail, not a second gate signal.

Trigger Conditions

Automatic — Record Phase (on every FH asset edit)

Whenever the 3-axis auto-gate in CLAUDE.md fires (any SKILL.md / rules / CLAUDE.md edit), append a manifest entry before committing.

Wiring note: This Record step is invoked manually or via harvest-loop until the CLAUDE.md 3-axis auto-gate is explicitly extended to call it. The auto-gate currently chains regression_guard → steel-quench → phantom-quench; adding edit-manifest Record as a pre-step is a proposed extension, not yet wired. Do not assume automatic invocation — call it explicitly after an FH asset edit.

Automatic — Verify Phase (harvest-loop Step 0-c)

At session start (via harvest-loop), read all pending entries and attempt verification.

Manual

Phrase	Action
"check edit manifest", "did our edits work?"	Full verify pass on pending entries
"show rejected edits", "what didn't work?"	Display rejected-edits buffer
"add to manifest", "record this edit"	Manual Record step

Execution Steps

RECORD mode (called on each FH asset edit)

Step R1 — Draft Entry

Prompt the AI (or the user, for high-stakes edits) to fill:

• edit_summary: one line, what changed
• predicted_impact: what observable improvement is expected
• predicted_measurable_by: how and when this can be checked

Step R2 — Append to Manifest

# Append YAML block to tracks/_meta/edit_manifest.yaml

Output confirmation: edit-manifest: entry em-{date}-{slug} recorded

Step R3 — Flag Untestable Predictions

If predicted_measurable_by is vague ("generally better", "feels cleaner") → mark validation_status: untestable immediately. These are tracked separately as a signal that the edit rationale needs sharpening.

---

VERIFY mode (harvest-loop Step 0-c or manual)

Step V1 — Load Pending Entries

# canonical pending + legacy freeform "predicted ..." entries (transition: reconcile legacy to pending)
# \b anchors the alternation so 'pending_review' / 'predicted_outcome' (non-canonical) still surface as legacy, not swept as pending
grep -nA22 -E 'validation_status: *"?(predicted|pending)\b' tracks/_meta/edit_manifest.yaml

Skip entries where predicted_measurable_by date has not yet passed. Reconcile any legacy freeform validation_status: "predicted — ..." entry to pending (move the prose into predicted_measurable_by) as you touch it — otherwise it stays invisible to future passes. Reconciliation completeness (a half-reconciled entry stays unverifiable): when you touch a legacy entry, also backfill the fields a canonical entry needs — generate a missing id (em-{date}-{slug}), set file: from the edit's target, and set validation_type explicitly (default mechanical when predicted_measurable_by names a grep/count/git check; judged when it names a reviewer observation). A missing validation_type is not a silent default — name it, or the entry can't be scored consistently across passes.

Step V2 — Verify Each Entry

First classify the entry by validation_type, then collect evidence accordingly:

• mechanical — prediction is a count/presence checkable by grep/git. Record baseline_value → measured_value → delta. The check IS the evidence (non-vacuous by construction).
• judged — prediction needs reviewer judgment. Requires one concrete cited observation (file:line / a quoted signal), never a bare "seems better". No citation → stays pending, not verified. Non-Model Ground (the citation must be tool-confirmed, not asserted): the cited file:line MUST be confirmed by an actual Grep/Read in this verify pass and the tool output (the matched line) pasted into verification_note — a citation-shaped string asserted from memory is NOT evidence. An unverifiable / un-pasted citation caps match_score at 0.5 (never 1.0). This is the same anchor discipline as phantom-quench: a verdict rests on a surfaced span, not a claim that one exists.
• untestable — no observable evidence source. Mark untestable, do not score.

For each entry, check the evidence source specified in predicted_measurable_by:

Evidence Source	Check Method
Session utterance patterns	Grep tracks/_meta/ for trigger phrases
Skill invocation count	Read knowledge/shared/learnings/subagent_invocations_log.yaml (create if absent)
User friction signals	Grep tracks/_meta/fh_signal_*.md for related friction
Git commit frequency	git log --oneline --since={date} -- {file}

Then score match_score: 1.0 = evidence clearly confirms the prediction · 0.5 = partial/ambiguous · 0.0 = contradicted or no-occurrence. Record the score + the cited evidence in verification_note.

Circularity guard: edit-manifest is invoked by harvest-loop (Step 0-c). To avoid a circular evidence loop, edit-manifest must NOT use harvest-loop's own synthesis outputs (proposal lists, curator decisions) as verification evidence. Evidence sources are limited to primary signals — raw utterance logs, invocation counts, git history, friction signals — that exist independently of any harvest-loop run.

Step V3 — Apply Validation Gate

match_score	status → Gate Decision	Next Action
≥ 0.75	verified → accepted	No action needed
0.25–0.75	verified(partial) → redefine	Sharpen the prediction/edit; note what partially held
≤ 0.25	falsified → rejected	Add to rejected-edits buffer; propose revert if regression
no evidence / window not matured	keep pending	Re-check next session
no evidence source	untestable	Flag for human judgment

Step V4 — Rejected-Edits Buffer Report

After verification pass, output rejected entries as a learning signal:

edit-manifest: verification complete
  Verified (accepted): N
  Falsified (rejected): N
  Still pending: N
  Untestable: N

Rejected edits (negative feedback buffer):
  em-2026-05-28-trigger-phrase: predicted "+1 session/week via phrase X"
    → falsified: no utterance evidence in 7 sessions
    → proposed revert: [y/N]

Step V5 — Update Manifest

Apply verified_at, verification_note, gate_decision to each resolved entry via Edit.

Validation Gate Logic

Mirrors SkillOpt's validation gate (the held-out gate over its selection-split score):

IF verified evidence shows improvement:
    gate_decision = accepted (edit stays)
ELSE IF falsified (no improvement or regression):
    gate_decision = rejected
    → retained in manifest as negative feedback
    → future edit proposals for same file grep rejected entries first
    → propose revert to human
ELSE:
    stay pending

Rejected-Edit Reuse: When proposing a new edit for a file, the AI must grep edit_manifest.yaml for prior rejected entries on that file and explicitly state why the new proposal avoids the same failure mode.

Constraints

• Append-only: Never delete manifest entries. Rejected edits are the most valuable learning signal — they prevent the same mistake twice.
• Human gate on revert: Gate decision rejected proposes revert; human confirms.
• Simplification guard: If no pending entries exist, output one line "edit-manifest: no pending entries" and exit.
• Untestable rate alarm: If untestable entries exceed 30% of total, flag: "edit rationale quality degrading — predictions are not falsifiable."

Done When

RECORD mode:
  Entry appended to edit_manifest.yaml
  + Untestable flag applied if vague prediction

VERIFY mode:
  All pending entries checked (canonical + legacy freeform reconciled to pending)
  + validation_type classified (mechanical / judged / untestable)
  + match_score recorded with cited evidence (mechanical: delta; judged: one cited observation)
  + Gate decisions applied (accepted / redefine / rejected / pending)
  + Rejected-edits buffer reported
  + Manifest file updated via Edit
  + Human gate presented for any proposed reverts

Check class (per harness_6axis_framework.md §Axis 5): the verify pass itself is measured for mechanical entries (delta is a number) and judged for judged entries — the judged path is kept non-vacuous by the mandatory cited observation (no citation → stays pending, never auto-verified).

References

• Theoretical basis: AHE (arXiv:2604.25850) §4 change manifest + prediction falsifiability
• Validation gate: SkillOpt (arXiv:2605.23904) §3.3 validation/held-out gate over the selection-split score + rejected-edit buffer
• Integrates with: harvest-loop Step 0-c · verify-bidirectional · 3-axis auto-gate (CLAUDE.md)
• Manifest file: tracks/_meta/edit_manifest.yaml