| name | derivation-audit |
| description | Audit the mi-spectral sourced derivation (manuscript/derivation.qmd, or any manuscript/*.qmd) for verification and citation-discipline compliance, and emit a flag-only report. READ-ONLY: it diagnoses and flags, it never edits the manuscript or literature. Use this whenever the user asks to audit, check, verify, vet, or review the derivation / the proof / the sourced derivation for citation integrity, broken source pointers, hallucinated or unsupported citations, unreproducible numbers, dangling cross-references, missing Monte Carlo asserts, or over-claiming — and when they say "/derivation-audit", "audit the derivation", "is every claim sourced", "check the proof's citations", or "does the evidence actually back the claims". |
Derivation audit
Audit a sourced derivation in manuscript/ against this repo's verification and
citation-discipline conventions, and produce a flag-only report. You
diagnose and flag; you never edit the manuscript, the literature/ corpus,
the .bib, or any todo//verification/ file. The deliverable is the report.
Target. The argument is the path to audit. Default to
manuscript/derivation.qmd (the canonical sourced derivation) when none is
given; also accept manuscript/mi-spectral-apa.qmd or any manuscript/*.qmd.
Why this skill exists
This is a citation-disciplined, human-prompted AI theory paper for JAIGP. The
load-bearing defense against the way human-prompted AI derivations fail in review is
that every external claim points to a real source passage that actually says
what the claim attributes to it, and every number points to a reproducible
artifact that exists. The repo enforces a slice of this at edit time (the
PreToolUse hook in .claude/hooks/check_citations.py blocks @citekey writes
with no backing literature/<citekey>.md). This skill is the standing,
whole-document audit the hook can't do: it mirrors the hook's mechanical rule
and checks the things the hook never sees — whether the source text supports
the claim, whether the cited numbers reproduce, whether the cross-references and
[mc] asserts hold together, and whether the prose over-claims its own scope.
How the document is built (the idioms you are auditing)
Read manuscript/derivation.qmd once to ground yourself before judging it. The
conventions it follows, and that your checks key off:
@citekey — a bibliographic citation. Must have BOTH a
literature/<citekey>.md (the Mathpix-converted source text, with YAML
frontmatter: citekey, authors, year, title, venue, doi, source_pdf, pdf_sha256, converted_via, last_synced) AND an entry in
literature/mi-spectral-better-bibTeX.bib. This is the hook's rule.@eq-… @sec-… @lst-… @fig-… @tbl-… @thm-… (and lem/cor/prp/cnj/def/exm/ exr/sol/rem) — Quarto cross-references, NOT citations. They resolve to a
{#label} defined in the doc (or a #| label: on a code cell). Never treat
these as citekeys — that is the classic false positive (the repo's own
lit_sync.py trips on it).@TODO:<slug> — a deliberate drafting placeholder that bypasses the hook.- "Source." margin notes (
::: {.column-margin}) — carry the provenance: a
source equation/theorem number, a verbatim snippet, and a :lineref of the
form <a href="#sec-src-<alias>"><code>:NNNN</code></a> pointing into the
literature/<citekey>.md copy. The snippet is a re-pagination-stable anchor.
- Step tags —
[derived] (proved here, not cited), [mc]
(machine-checked: additionally has an executable assert in Appendix B),
[cited]. Definition/regularity/lemma labels are D-, R-, L-, A-.
- Appendix A (
#sec-src-*) — verbatim source passages grouped by reference,
each > quoted from the literature/ copy with its equation/theorem number
and line.
- Appendix B (
#sec-verify) — inline ```{python} cells labelled
#| label: lst-mc-*, each reducing lhs - rhs to 0 symbolically or
asserting a match to verification/00-setup.R to machine precision. Every
[mc] tag should correspond to one of these with a real assert.
- Reproducibility layering — numbers trace to
verification/run_all.R
phases (outputs under verification/cache/run_all-<suffix>/phase{0..N}/),
standalone verification/*.R diagnostics, or committed *.rds/summary.csv
caches. The originals W1-bias-decomposition.R etc. are preregistered;
_legacy/ scripts are retired and must not be cited as live artifacts.
Step 1 — run the mechanical engine
Run the bundled script from the repo root. It does the deterministic checks
(citation backing, crossref resolution, reproducibility-pointer existence,
[mc]↔assert pairing, lineref resolution, Appendix-A snippet fingerprints,
todo/003 lifecycle) and prints a markdown report plus a JSON dump.
py .claude/skills/derivation-audit/scripts/audit_derivation.py <target.qmd> --json <tmp>/derivation-audit.json
(Rscript is not involved; this is pure Python, stdlib only.) Read both the
markdown and the JSON. The JSON summary block gives you the working lists you
need for Step 2: cites.citekeys, mc.listings, alias_map
(sec-src-<alias> → citekey), and the resolved/total lineref counts.
The engine is the source of truth for dimensions A (citation integrity),
D (crossref resolution), C (reproducibility pointers + [mc] asserts),
F (lifecycle), and the mechanical half of B (linerefs resolve;
snippet fingerprints). Carry its findings into your report verbatim (adjust
severity only with a stated reason). Do not re-derive what it already
computed.
Step 2 — the semantic source check (dimension B, the hard part)
This is the check the engine cannot do and the reason the skill exists: does
the cited source actually support the claim made about it? A citation that
resolves to a real file and a real line can still attribute an equation the
source never states, or assert something the source contradicts. Mirror the
repo rule: reliance on training-weight knowledge of a cited paper is
prohibited — verify against the local text.
For each cited @citekey (use cites.citekeys from the JSON), and for each
distinct claim the derivation attributes to it (the margin "Source." notes and
the Appendix-A entries name the equation/theorem number and the proposition the
derivation leans on):
- Read the relevant span of
literature/<citekey>.md (use the :lineref
anchors and the Appendix-A line numbers to jump to the right place — you do
not need the whole file).
- Confirm the source contains the cited equation/theorem/result, and that
the claim the derivation makes about it is supported by the source text —
not merely that the paper is cited near the claim.
- Confirm each Appendix-A
> verbatim block is genuinely present (exact or
near-exact, modulo Mathpix math markup) in that file. The engine flags
fingerprint misses as INFO; you make the call on whether the quote is
faithful or drifted.
Grade each attribution:
- supported — no finding.
- mismatch / unsupported / unverifiable from the local text — BLOCKER.
This is the user's "evidence tangibly validated against external exact text":
a claim whose attributed source does not back it fails the audit.
Also flag (here, where you have the source in hand):
- An external or non-trivial assertion that carries no source pointer and is
not marked
[derived] — WARN (an un-sourced claim that reads as if
external).
- A step marked
[derived] that secretly leans on an external result —
WARN (it should cite, or the derivation should actually be self-contained).
Bound the work. This can fan out. If a subagent tool is available, dispatch
one reader per cited citekey (or per source-section), each returning a short
verdict list {claim, source eq/thm, supported|mismatch|unverifiable, evidence}; then fold the verdicts into your report. Otherwise read the sources
inline, prioritizing the load-bearing citations (the ones the theorem and the
two applications actually turn on — e.g. the missing-information identity, the
RIV identification, the variance-recovery lemma, the observed-information-under-
MAR choice, and whatever the headline-IC positioning rests on) over the
boilerplate matrix-identity citations.
Step 3 — evidence-type vs claim-type (dimension C, semantic)
Beyond "does the artifact exist" (engine), check that the kind of evidence
matches the kind of claim:
- An "exact" / "symbolic" / "parameterization-free" claim backed only by
Monte Carlo or a single numeric replicate → WARN (evidence weaker than the
claim).
- An
O(1) / asymptotic / "does not decay" / O(n^-1) claim backed only
by a single finite-n point → WARN.
- A number or "the simulation confirms / the cache gives" with no
reproducible pointer at all → WARN (the engine catches broken pointers; you
catch missing ones in prose).
Step 4 — internal consistency of the headline (dimension D, semantic)
The engine confirms cross-refs resolve. You additionally confirm the headline
result is stated consistently across the introduction, the theorem statement,
and the combine step — same governing equation, same scope conditions. Flag
contradictions (WARN), e.g. an MCAR-only clean result stated as general in one
place and scoped in another, or the (A)+(C) design-imbalance term present in
one statement and silently dropped in another.
Step 5 — honesty / over-claiming (dimension E)
- Flag language that outruns its stated verification — "proved", "exact",
"for any regular family", "general", "holds in general" — wherever the doc's
own scope notes say "asserted", "spot-checked", "to leading order",
"
O(n^-1)", "open", "MVN instantiation", "bivariate-monotone only", or
"verified empirically (symbolic check deferred)". Surface any claim stronger
than the evidence the same document admits. Severity: WARN (BLOCKER only
if the overclaim is the headline itself).
- Flag house-jargon in outside-reader-facing prose (intro, roadmap, proof
sketch, motivated narrative):
ledger, numerical witness, tagged,
machine-checked, Stage N, consume. Severity: INFO. (These are fine
in margin notes and internal scaffolding; the ban is about reader-facing
prose.)
Output — one flag-only markdown report
Print a single report. Begin with a one-line PASS / FAIL summary (FAIL iff
any BLOCKER) and per-dimension counts, then group findings by severity. Each
finding is one line:
<file>:<line> — <issue> — evidence: <found | "snippet not in source" | "no cache" | …> — fix: <one-line suggested fix>
Severity:
- BLOCKER — a missing
literature/<citekey>.md for a real citekey (the
hook's rule); a claim whose attributed source does not support it or cannot be
verified from the local text; a lineref that overruns its source file; an
[mc]/@lst-mc-* reference with no backing listing. These fail the audit.
- WARN — over-claiming beyond stated scope; evidence-type vs claim-type
mismatch; un-sourced non-trivial assertion;
[mc] listing with no assert;
reproducibility pointer to a missing or _legacy/ artifact; dangling
crossref; headline inconsistency; cited-but-not-logged-as-verified in
todo/003; @TODO: placeholder; citekey missing from the .bib.
- INFO — house-jargon in reader-facing prose; Appendix-A snippet fingerprint
misses you judged to be markup artifacts rather than real drift; foundational
corpus not in the todo/003 lifecycle (tracked in todo/001).
Suggested-fix lines describe the fix; do not apply it. Use the template
below.
## Derivation audit — <file>
**<PASS|FAIL>** — <nB> BLOCKER / <nW> WARN / <nI> INFO
A citation <a/b/c> · B evidence↔source <…> · C reproducibility <…> · D consistency <…> · E honesty <…> · F lifecycle <…>
### BLOCKER (<n>)
- <file>:<line> — <issue> — evidence: <…> — fix: <…>
### WARN (<n>)
- …
### INFO (<n>)
- …
Guardrails
- Read-only. Never Edit/Write/MultiEdit the manuscript,
literature/, the
.bib, or todo//verification/ files. If asked to fix something, propose
the fix in the report and stop. (Writing this report to a scratch file the
user asked for is fine; touching the audited material is not.)
- No training-weight citations. Every B verdict must rest on text you read
in
literature/<citekey>.md this session. If a source file is missing, that
is the BLOCKER — do not paper over it from memory.
- Don't re-flag what the engine flagged; fold its findings in and add the
semantic ones. If you disagree with an engine severity, say so with a reason.
- Use Windows-style backslash paths with the file tools (per the repo's
CLAUDE.md), and run the script with py from the repo root.
