retrospective

name	retrospective
description	Phase 9 retrospective — read a window or set of recently-merged PRs, classify each harness/run finding with the eval-failure taxonomy, and append tagged learnings to docs/retro/learnings.md. Propose-only — writes ONLY under docs/retro/. Use when explicitly asked to "run a retrospective", "retro the last N PRs", or to review a batch of merged work for harness lessons. Do not auto-run.

Retrospective (Phase 9)

Reads the run record Butler already emits and captures tactical learnings about the harness — the gotchas, misparses, and recurring failure shapes that a batch of work surfaced. This is the manual entry point (/retrospective) for the continual-improvement pillar; see docs/adr/0005-retrospective-agent.md.

Scope: capture learnings, emit amendment proposals, and escalate recurring failure modes by filing P2 ready-for-agent issues automatically when the 3+-same-mode threshold is crossed.

The one hard rule

Writes ONLY under docs/retro/. Never edit CLAUDE.md, a skill, .claude/settings.json, or any other harness surface — the retrospective proposes, a human applies. Writes:

• Appending to docs/retro/learnings.md
• Emitting prose proposal files under docs/retro/amendments/
• Appending golden-case candidates to docs/retro/candidates.jsonl

Amendments are prose, not diffs; nothing is auto-applied. Opening a GitHub issue is the only side-effect that reaches outside docs/retro/, and it is scoped to the escalation step below.

When to run

• An explicit request: "run a retrospective", "retro the last N PRs / since ".
• Automatically, at the end of every /implement-queue batch, over the PRs the batch produced.

Do NOT run spontaneously. It runs only on explicit invocation.

Step 1 — Establish the window

Resolve what to review from the argument: a count ("last 5 PRs"), a git ref/date ("since v0.3", "since 2026-06-01"), or an explicit PR list. Default to the PRs merged since the last retrospective run if unspecified.

gh pr list --state merged --limit <N> --json number,title,mergedAt,url

Step 2 — Read the run record

For each PR in the window, gather the signal Butler already emits — no new instrumentation:

• the merged diff and commit messages (gh pr diff <n>, gh pr view <n>)
• CI/check outcomes and any re-runs (gh pr checks <n>)
• /review findings recorded on the PR (review comments)
• git history around the merge

claude-mem observations are optional enrichment — they're per-user and may be absent in headless runs, so never depend on them.

Step 3 — Classify each finding

For every genuine harness/run finding, assign exactly one failure mode from the eval-failure taxonomy (FailureModeSchema in packages/evals/src/schemas/golden-case.ts; prose in docs/eval-failure-taxonomy.md):

infinite_tool_loop, hallucinated_data, silent_error_swallowing, tool_output_misparse, config_schema_drift, timezone_locale_mishandling, unsafe_destructive_action, spec_deviation.

One shared vocabulary with the eval harness — never invent a parallel label. A clean batch with no real findings produces no learnings; do not manufacture them.

Step 4 — Append tagged learnings

Append one bullet per finding to docs/retro/learnings.md, in the exact format the learnings-store module parses:

- [<failure_mode>] <ISO-8601 timestamp> — <one-line gotcha> (source: <PR #, commit, or session>)

Keep the gotcha to one tactical line; cite the concrete source. Do not hand-edit or reorder existing entries — the log is append-only.

Step 5 — Emit amendment proposals

Some findings don't just record a gotcha — they imply a harness change (a CLAUDE.md rule, a skill step, a taxonomy entry, an ADR, a golden case, or an issue). For each such finding, emit one prose proposal file via the amendment-writer module (packages/evals/src/retro/amendment-writer.ts):

import { createAmendmentWriter } from "@butler/evals";

const writer = createAmendmentWriter("docs/retro/amendments");
writer.write(
  {
    title: "<one-line finding>",
    target_surface: "<CLAUDE.md | a skill | the taxonomy | an ADR | a golden case | an issue>",
    failure_mode: "<one taxonomy mode>",
    recommended_change: "<the change you recommend, in prose>",
    evidence: ["PR #…", "commit …"], // optional — omit if there's nothing concrete to cite
  },
  "<YYYY-MM-DD>", // injected date, e.g. today
);

This writes one docs/retro/amendments/{date}-{slug}.md per harness-relevant finding. The file names a target surface and the exact edit it recommends, as prose a human reads and applies (or rejects) — never a diff, never auto-applied. A finding with no evidence still renders a well-formed file ("None provided").

A clean run produces no amendment files. Only emit a proposal for a finding that genuinely implies a harness change; do not manufacture proposals from a clean batch.

Step 6 — Escalate recurring failure modes

After all learnings are appended (Step 4), run the retro-escalate CLI to detect modes that have crossed the 3+-occurrence threshold and file a P2 ready-for-agent GitHub issue for each new escalation. Idempotency is built in: already-escalated modes are tracked in docs/retro/candidates.jsonl, so re-running the skill never re-files an issue for a mode that has already been escalated.

# Build evals to ensure dist/cli/retro-escalate.js is fresh
pnpm --filter @butler/evals build

# Run from the project root so relative paths (docs/retro/) resolve correctly.
# Each line of stdout is a JSON issue payload: { title, body, labels[] }
node packages/evals/dist/cli/retro-escalate.js

For each line the CLI prints to stdout (one per new escalation), file a GitHub issue:

# Parse the JSON and call gh issue create. Use a temp file to handle multi-line bodies.
PAYLOAD='<the JSON line from stdout>'
TITLE=$(echo "$PAYLOAD" | node -e "process.stdin.resume();let b='';process.stdin.on('data',d=>b+=d);process.stdin.on('end',()=>console.log(JSON.parse(b).title))")
BODY=$(echo "$PAYLOAD" | node -e "process.stdin.resume();let b='';process.stdin.on('data',d=>b+=d);process.stdin.on('end',()=>console.log(JSON.parse(b).body))")
gh issue create --title "$TITLE" --body "$BODY" --label "P2" --label "ready-for-agent"

In practice, use the Bash tool to pipe the CLI output through while IFS= read -r line and call gh issue create for each non-empty line. A clean run (no threshold crossings, or all modes already escalated) produces no stdout and no issues — that is the normal outcome.

Idempotency guarantee: the CLI reads docs/retro/candidates.jsonl on every invocation. A mode that already has a candidate in that file is excluded from the detector's output, so re-running the full skill over the same learnings never fires a duplicate issue.

Step 7 — Report

Summarise: window reviewed, learnings captured (with their modes), amendment proposals emitted (one line each, naming the target surface), issues filed for escalations (with issue URLs), and modes whose candidates are now in docs/retro/candidates.jsonl.