haipipe-paper-claim-audit

name	haipipe-paper-claim-audit
description	Zero-context verification that every number, comparison, and scope claim in the paper matches raw result files. Uses a fresh cross-model reviewer with NO prior context to prevent confirmation bias. Use when user says "审查论文数据", "check paper claims", "verify numbers", "论文数字核对", or before submission to ensure paper-to-evidence fidelity.
argument-hint	[paper-directory]
allowed-tools	Bash(*), Read, Write, Edit, Grep, Glob, Agent, mcp__codex__codex
metadata	{"version":"0.1.1","last_updated":"2026-05-31","summary":"Zero-context verification that every number, comparison, and scope claim in the paper matches raw result files."}

Paper Claim Audit: Zero-Context Evidence Verification

Verify that every claim in the paper matches raw evidence for: $ARGUMENTS

Why This Exists

The executor writes experiments AND writes the paper. It "knows" what the results should be. This creates confirmation bias:

• Rounding 84.7% up to 85.3%
• Reporting best seed instead of average
• Citing metrics from a different experiment config
• Claiming "improves by 15%" when the delta is actually 12.8%

A fresh reviewer with zero prior context catches these because it has no expectations — it just compares paper text vs raw files.

How This Differs From Other Audit Skills

Skill	Question it answers
/probe-audit	Is the experiment code honest? (fake GT, normalization fraud)
/result-to-claim	Does the data scientifically support this claim?
/haipipe-paper-claim-audit	Does the paper report the data truthfully and precisely?

Core Principle

Zero-context, fresh reviewer. The auditor receives ONLY:

• Paper .tex files (the claims)
• Raw result files (the evidence)

It does NOT receive:

• ❌ EXPERIMENT_LOG.md
• ❌ EXPERIMENT_TRACKER.md
• ❌ AUTO_REVIEW.md
• ❌ NARRATIVE_REPORT.md
• ❌ Any executor summary or interpretation
• ❌ Any prior audit results
• ❌ Any conversation history

This is stricter than reviewer-independence — it's zero-context evidence audit.

Workflow

Step 1: Collect Files (Executor — Claude)

Locate paper and result files WITHOUT reading or interpreting them.

Paper files (claims) — paths shown relative to the shell's working directory so you can find them with ls; when writing them into audited_input_hashes, use paths relative to the paper dir (no paper/ prefix) per the "Submission Artifact Emission" section below:

paper/main.tex                # → hash key: main.tex
paper/sections/*.tex          # → hash key: sections/*.tex
paper/tables/*.tex (if separate)   # → hash key: tables/*.tex

Result files (evidence):

results/*.json, results/*.jsonl, results/*.csv, results/*.tsv
outputs/*.json, outputs/*.csv
wandb-summary.json (if exists)
**/metrics.json, **/eval_results.json
**/config.yaml, **/args.json (experiment configs)

Exclude (no summaries, no interpretations):

EXPERIMENT_LOG.md, EXPERIMENT_TRACKER.md, AUTO_REVIEW*.md
NARRATIVE_REPORT.md, PAPER_PLAN.md, findings.md
Any .md file that is an executor-written summary

Step 2: Fresh Reviewer Audit (GPT-5.4 — NEW thread, no reply)

CRITICAL: Use mcp__codex__codex (new thread), NEVER mcp__codex__codex-reply. Every run must be a fresh context.

The full auditor prompt -- the audit protocol (A extract every quantitative claim -> B trace each to evidence -> C the 7 failure-mode checks), the per-claim output schema, and the PASS/WARN/FAIL verdict -- lives in ref/auditor-prompt.md. Dispatch it verbatim as a fresh mcp__codex__codex thread (model gpt-5.4, model_reasoning_effort: xhigh), filling in the listed paper .tex paths and raw result-file paths.

Step 3: Write Report (Executor — Claude)

Parse the reviewer's response and write PAPER_CLAIM_AUDIT.md:

# Paper Claim Audit Report

**Date**: [today]
**Auditor**: GPT-5.4 xhigh (fresh zero-context thread)
**Paper**: [paper title from tex]

## Overall Verdict: [PASS | WARN | FAIL]

## Claims Verified: [N total]
- exact_match: [count]
- rounding_ok: [count]
- ambiguous_mapping: [count]
- missing_evidence: [count]
- mismatch: [count]

## Issues Found

### [FAIL/WARN] Claim #N: [description]
- **Location**: Section X / Table Y / Figure Z
- **Paper says**: "..."
- **Evidence shows**: ...
- **Status**: [status]
- **Fix**: [specific correction needed]

## All Claims (detailed)

| # | Location | Paper Value | Evidence Value | Status |
|---|----------|-------------|---------------|--------|
| 1 | Table 2 | 85.3% | 85.28% | rounding_ok |
| 2 | Abstract | "15% improvement" | 12.8% | number_mismatch |
| ... |

Also write PAPER_CLAIM_AUDIT.json for machine consumption.

Step 4: Print Summary

📋 Paper Claim Audit Complete

  Claims verified: 24
  exact_match:     18
  rounding_ok:      3
  ambiguous:         1
  ⚠️ mismatch:      2

  Overall: ⚠️ WARN

  See PAPER_CLAIM_AUDIT.md for details.

When to Run

1. After /haipipe-paper section-edit — first check once prose is drafted
2. After a revise/polish pass — recheck if edits changed any numbers
3. Before submission — final verification

Integration with Other Skills

Advisory, Never Blocking

Same pattern as /probe-audit:

• PASS → continue normally
• WARN → print warning, continue, flag draft as "check numbers before submission"
• FAIL → print alert, continue, but do NOT mark as submission-ready

Review Tracing

After each mcp__codex__codex or mcp__codex__codex-reply reviewer call, save the trace following shared-references/review-tracing.md. Use tools/save_trace.sh or write files directly to .aris/traces/<skill>/<date>_run<NN>/. Respect the --- trace: parameter (default: full).

Submission Artifact Emission

This skill always writes paper/PAPER_CLAIM_AUDIT.json, regardless of caller or detector outcome. A detector-negative run (paper has no numeric claims) emits verdict NOT_APPLICABLE; a paper-with-numeric-claims-but-no-raw-results run emits BLOCKED. Silent skip is forbidden — paper-writing Phase 6 and tools/verify_paper_audits.sh both rely on this artifact existing at a predictable path.

The artifact conforms to the schema in shared-references/assurance-contract.md:

{
  "audit_skill":      "haipipe-paper-claim-audit",
  "verdict":          "PASS | WARN | FAIL | NOT_APPLICABLE | BLOCKED | ERROR",
  "reason_code":      "all_numbers_match | rounding_drift | missing_raw_results | ...",
  "summary":          "One-line human-readable verdict summary.",
  "audited_input_hashes": {
    "main.tex":                              "sha256:...",
    "sections/5.evidence.tex":               "sha256:...",
    "/abs/path/to/results/run_2026_04_19.json": "sha256:..."
  },
  "trace_path":       ".aris/traces/haipipe-paper-claim-audit/<date>_run<NN>/",
  "thread_id":        "<codex mcp thread id>",
  "reviewer_model":   "gpt-5.4",
  "reviewer_reasoning": "xhigh",
  "generated_at":     
   
       <int>
          ... per-claim issue records ... 
        ... raw files consulted ... 
  

audited_input_hashes scope

Hash the declared input set passed into this audit invocation — i.e. the exact .tex files and raw result / config files this run read — not a repo-wide union and not the reviewer's self-reported subset. If a caller passed only main.tex + a single result file, hash those two files and no others. The external verifier rehashes these entries; any mismatch flags STALE.

Path convention (must match what tools/verify_paper_audits.sh expects): keys are paths relative to the paper directory (the arg passed to the verifier) for in-paper files — so main.tex, not paper/main.tex — and absolute paths for out-of-paper files such as external results/ dirs. The verifier resolves relative entries via os.path.join(paper_dir, key); prefixing with paper/ produces paper/paper/main.tex and false-fails as STALE.

Verdict decision table

Input state	Verdict	reason_code example
No numeric claims detected in paper	NOT_APPLICABLE	no_numeric_claims
Numeric claims detected, no raw result files found	BLOCKED	no_raw_evidence
All claims reconcile to raw data	PASS	all_numbers_match
Minor rounding drift only, no material mismatch	WARN	rounding_drift
Any material mismatch (wrong number, config mismatch)	FAIL	claim_mismatch
Reviewer invocation failed (network / malformed)	ERROR	reviewer_error

Thread independence

Every invocation uses a fresh mcp__codex__codex thread. Never codex-reply. Do not accept prior audit outputs (PROOF_AUDIT, CITATION_AUDIT, EXPERIMENT_LOG, AUTO_REVIEW summaries) as input to this audit — the fresh thread preserves reviewer independence per shared-references/reviewer-independence.md.

Human-readable sibling

paper/PAPER_CLAIM_AUDIT.md is written alongside the JSON for readers. The JSON is authoritative for tools/verify_paper_audits.sh; the Markdown is for humans. The parent skill (paper-writing Phase 6) plus the verifier decide whether the verdict blocks finalization — this skill itself never blocks; it only emits.