kookr-rfc-critic-meta-analysis

name	kookr-rfc-critic-meta-analysis
description	Kookr-internal extension to rfc-iterative-review that captures append-only critic traces for later meta-analysis of RFC reviewer subagents.
keywords	kookr, rfc, critic, subagent, meta-analysis, reviewer, trace, iterative-review, prompt evolution
related	rfc-iterative-review, kookr-skill-naming-convention

RFC Critic Meta-Analysis

Use this skill after rfc-iterative-review when drafting or revising Kookr RFCs that should contribute evidence about reviewer-subagent quality.

This skill does not replace rfc-iterative-review. It extends that workflow with a lightweight trace so future reports can answer which critics were useful, noisy, missing risks, or worth changing.

Scope

This is Kookr-internal because it writes Kookr RFC artifacts under docs/rfc/ and informs Kookr's reviewer-subagent roster. Do not promote it to the toolkit plugin unless the paths and schema ownership are generalized.

For the design rationale, see docs/rfc/rfc-subagent-feedback-meta-analysis.md.

Trace Location

For an RFC at:

docs/rfc/rfc-<slug>.md

write the trace at:

docs/rfc/meta/rfc-<slug>.critic-trace.jsonl

Each line is one JSON object using docs/schemas/critic-trace.v1.json.

When to Capture

During RFC review:

1. Run the normal rfc-iterative-review critic rounds.
2. After each critic returns, append one invocation event.
3. After the convergence consensus attack returns, append one invocation event too. It fires once, on the way out of Phase 2, and is not a panel lens — but its outcome is exactly what this trace measures, so it must not go unrecorded. Set critic: "general-purpose", extractionNote: "consensus-attack at convergence; not a panel lens", and reuse the final round number for round (the schema requires an integer ≥ 1, and a convergence-time run has no round of its own). Set noSubstantiveFindings: true when the verdict is "consensus survives" — a null verdict is a recorded outcome, not a skipped run.
4. For each substantive finding that survives triage, append one finding event.
5. Keep the normal human-readable Critic Feedback Incorporated section in the RFC.
6. At the end of the RFC, add a short Meta-analysis readiness note in the critic-feedback section.

Do not record every sentence from critic output. Record substantive findings: defects, missing constraints, empirical-check requests, scope changes, user-intent questions, or useful simplifications.

Minimal Events

Invocation

Write one invocation event per critic per round, including critics with no substantive findings, plus one for the convergence consensus attack (see "When to Capture" step 3).

{
  "schemaVersion": "critic-trace.v1",
  "recordType": "invocation",
  "rfcPath": "docs/rfc/rfc-example.md",
  "round": 1,
  "critic": "failure-mode-analyst",
  "sourceRef": {
    "kind": "manual_excerpt",
    "excerptHash": "sha256:<hash>"
  },
  "substantiveFindingsExtracted": true,
  "extractedFindingIds": ["rfc-example-r1-failure-mode-analyst-f1-1a2b3c4d"],
  "noSubstantiveFindings": false,
  "extractionNote": "Raw subagent transcript unavailable; finding copied from final subagent response."
}

If the critic produced no substantive finding, use:

{
  "schemaVersion": "critic-trace.v1",
  "recordType": "invocation",
  "rfcPath": "docs/rfc/rfc-example.md",
  "round": 2,
  "critic": "design-minimalist",
  "sourceRef": { "kind": "unavailable" },
  "substantiveFindingsExtracted": true,
  "extractedFindingIds": [],
  "noSubstantiveFindings": true
}

Finding

Generate findingId as:

<rfc-slug>-r<round>-<critic>-f<ordinal>-<8-char-rawFindingHash>

Example:

{
  "schemaVersion": "critic-trace.v1",
  "recordType": "finding",
  "rfcPath": "docs/rfc/rfc-example.md",
  "round": 1,
  "critic": "failure-mode-analyst",
  "findingId": "rfc-example-r1-failure-mode-analyst-f1-1a2b3c4d",
  "summary": "Outcome updates must be append-only, not mutable fields.",
  "rawFinding": "Delayed outcome updates can corrupt historical records if old JSONL lines are rewritten.",
  "sourceRef": {
    "kind": "manual_excerpt",
    "excerptHash": "sha256:1a2b3c4d..."
  },
  "disposition": "used",
  "dispositionReason": "RFC changed to append outcome events separately from finding events.",
  "valueNote": "Preserves time-horizon separation and auditability."
}

For duplicates, preserve the later critic's record and point at the first matching finding:

{
  "schemaVersion": "critic-trace.v1",
  "recordType": "finding",
  "rfcPath": "docs/rfc/rfc-example.md",
  "round": 1,
  "critic": "socratic-challenger",
  "findingId": "rfc-example-r1-socratic-challenger-f2-9e8d7c6b",
  "summary": "Outcome ownership is unclear.",
  "rawFinding": "Who is allowed to assign laterOutcome, and under what evidence standard?",
  "sourceRef": {
    "kind": "manual_excerpt",
    "excerptHash": "sha256:9e8d7c6b..."
  },
  "disposition": "duplicate",
  "dispositionReason": "Same issue as failure-mode finding f1, with different framing.",
  "duplicateOfFindingId": "rfc-example-r1-failure-mode-analyst-f1-1a2b3c4d",
  "duplicateContext": "parallel_independent"
}

Outcome

Append outcomes later. Do not edit the original finding line.

{
  "schemaVersion": "critic-trace.v1",
  "recordType": "outcome",
  "findingId": "rfc-example-r1-failure-mode-analyst-f1-1a2b3c4d",
  "assessedAt": "2026-05-13T19:00:00Z",
  "assessor": "implementer",
  "horizon": "post_pr_open",
  "outcome": "validated",
  "evidence": "The implementation needed a separate outcome event to avoid rewriting original finding events.",
  "evidenceRefs": [
    {
      "kind": "pr",
      "ref": "https://github.com/kookr-ai/kookr/pull/123",
      "summary": "PR implemented append-only outcome events."
    }
  ]
}

Missed Risk

Do not force missed-risk bookkeeping during drafting. When producing an aggregate report, either append missed_risk events for the horizon being evaluated or mark the report's trace-completeness section as missed-risk sweep not performed.

{
  "schemaVersion": "critic-trace.v1",
  "recordType": "missed_risk",
  "rfcPath": "docs/rfc/rfc-example.md",
  "discoveredAt": "2026-05-14T10:00:00Z",
  "evidence": "PR review found the schema allowed orphaned outcome events.",
  "expectedCritics": ["failure-mode-analyst", "boundary-critic"],
  "matchedFindingIds": [],
  "note": "No round-1 critic identified orphaned outcome validation."
}

Bias Controls

Minimum controls for every trace:

• Preserve raw finding text or a hashed excerpt pointer.
• Preserve one invocation event per critic run, including explicit no-finding runs.
• Preserve rejected, duplicate, and falsified findings.
• Do not use incorporation rate alone.
• Include author-bias context in dispositionReason when rejection depends on user intent, scope budget, or subjective taste.
• Require human approval for every prompt or roster diff.

Before broad/default rollout of a critic change:

• Evaluate a holdout set not used to motivate the change.
• Use a blinded finding packet where feasible.
• Compare against false-positive burden and missed-risk examples, not only useful findings.
• Validate across at least two time windows or RFC types before retiring a critic or changing default roster rules.

Evolution Paths

Fast Path: Low-Risk Prompt Clarification

Allowed for narrow clarifications that preserve a critic's role.

Required packet:

• One concrete trace example.
• One narrow prompt diff.
• One counterexample check explaining why the diff is not RFC-specific.
• One rollback condition describing what future trace pattern should revert it.
• Human approval.

Do not claim global improvement until later traces confirm it.

Full Path: Prompt Philosophy or Roster Change

Required for adding, retiring, splitting, merging, or substantially changing a critic.

1. Observe at least five RFCs or two active RFC-writing days, and at least three invocations of the target critic.
2. Diagnose with a manual aggregate report.
3. Propose a promotion packet with target dimensions, diff, examples, holdout plan, expected failure mode, and rollback condition.
4. Trial in shadow mode or on holdout RFCs.

Do not retire high-variance critics solely because their useful findings are infrequent. design-minimalist and ambition-amplifier are adversarial by design; a critic can be valuable even when the author chooses the other side.

Manual Aggregate Report

After 3-5 complete traces, write:

docs/reports/rfc-critic-meta-analysis-YYYY-MM-DD.md

Include:

• Corpus: RFCs, rounds, critics, total substantive findings.
• Trace completeness: raw finding coverage, missing source refs, invalid events, orphaned outcomes, duplicate IDs, unknown/expired outcomes.
• High-value finding examples.
• False-positive examples.
• Missed-risk examples, or missed-risk sweep not performed.
• Adversarial-pair outcomes.
• Cost notes: triage burden, extra rounds, and visible model/runtime cost.
• Proposed changes, or insufficient evidence.
• Holdout/trial recommendation if a full-path change is proposed.

Validation Before Aggregate Use

Before using a trace in an aggregate report:

• Count and exclude invalid JSONL lines.
• Count and exclude duplicate findingId values from prompt-change evidence.
• Count and exclude orphaned outcome events.
• Count and exclude events with unknown enum values.
• Treat missing required fields as invalid.

Invalid events are evidence of trace-quality problems, not evidence about critic quality.