result-to-claim

name	result-to-claim
description	Use when experiments complete to judge what claims the results support, what they don't, and what evidence is still missing. Codex MCP evaluates results against intended claims and routes to next action (pivot, supplement, or confirm). Use after experiments finish — before writing the paper or running ablations.
argument-hint	["experiment-description-or-wandb-run"]
allowed-tools	Bash(*), Read, Grep, Glob, Write, Edit, mcp__codex__codex, mcp__codex__codex-reply

Result-to-Claim Gate

🔒 Do not wrap this skill in /loop, /schedule, or CronCreate. It is verdict-bearing — it judges whether results support a claim. Re-running that verdict on a wall-clock timer adds no new signal (the verdict changes only when the results change, not when the clock ticks). What you actually want to schedule is the external wait that precedes it — experiments done → then run this gate once. See shared-references/external-cadence.md.

Experiments produce numbers; this gate decides what those numbers mean. Collect results from available sources, get a Codex judgment, then auto-route based on the verdict.

Context: $ARGUMENTS

When to Use

• After a set of experiments completes (main results, not just sanity checks)
• Before committing to claims in a paper or review response
• When results are ambiguous and you need an objective second opinion

Workflow

Step 1: Collect Results

Gather experiment data from whatever sources are available in the project:

1. W&B (preferred): wandb.Api().run("<entity>/<project>/<run_id>").history() — metrics, training curves, comparisons
2. EXPERIMENT_LOG.md: full results table with baselines and verdicts
3. EXPERIMENT_TRACKER.md: check which experiments are DONE vs still running
4. Log files: ssh server "tail -100 /path/to/training.log" if no other source
5. docs/research_contract.md: intended claims and experiment design

Assemble the key information:

• What experiments were run (method, dataset, config)
• Main metrics and baseline comparisons (deltas)
• The intended claim these experiments were designed to test
• Any known confounds or caveats

Step 1.5: Evidence Pre-Check

Before asking a model to judge support, mechanically verify that each cited value + source pair exists. Resolve tools/evidence_check.py via the usual .debuffer_skills/tools/ → tools/ → repo-root chain, then run it on a JSON batch of claim evidence pairs when available:

python3 "$EVIDENCE_CHECK" . --batch result_to_claim_evidence.json

verified only means the cited file and value exist; it does not mean the claim is supported. path_missing or value_not_found downgrades confidence and must be included in the Codex prompt as a blocker or caveat. See shared-references/evidence-precheck.md.

Step 2: Codex Judgment

Send the collected results to Codex for objective evaluation:

mcp__codex__codex:
  config: {"model_reasoning_effort": "xhigh"}
  prompt: |
    RESULT-TO-CLAIM EVALUATION

    I need you to judge whether experimental results support the intended claim.

    Intended claim: [the claim these experiments test]

    Experiments run:
    [list experiments with method, dataset, metrics]

    Results:
    [paste key numbers, comparison deltas, significance]

    Baselines:
    [baseline numbers and sources — reproduced or from paper]

    Known caveats:
    [any confounding factors, limited datasets, missing comparisons]

    Please evaluate:
    1. claim_supported: yes | partial | no
    2. what_results_support: what the data actually shows
    3. what_results_dont_support: where the data falls short of the claim
    4. missing_evidence: specific evidence gaps
    5. suggested_claim_revision: if the claim should be strengthened, weakened, or reframed
    6. next_experiments_needed: specific experiments to fill gaps (if any)
    7. confidence: high | medium | low

    Be honest. Do not inflate claims beyond what the data supports.
    A single positive result on one dataset does not support a general claim.

Step 3: Parse and Normalize

Extract structured fields from Codex response:

- claim_supported: yes | partial | no
- what_results_support: "..."
- what_results_dont_support: "..."
- missing_evidence: "..."
- suggested_claim_revision: "..."
- next_experiments_needed: "..."
- confidence: high | medium | low

Step 3.5: Check Experiment Integrity (if audit exists)

Skip this step if EXPERIMENT_AUDIT.json does not exist.

if EXPERIMENT_AUDIT.json exists:
    read integrity_status from file
    attach to verdict output:
        integrity_status: pass | warn | fail

    if integrity_status == "fail":
        append to verdict: "[INTEGRITY CONCERN] — audit found issues, see EXPERIMENT_AUDIT.md"
        downgrade confidence to "low" regardless of Codex judgment

    if integrity_status == "warn":
        append to verdict: "[INTEGRITY: WARN] — audit flagged potential issues"
else:
    integrity_status = "unavailable"
    verdict is labeled "provisional — no integrity audit run"
    (this does NOT block anything — pipeline continues normally)

See shared-references/experiment-integrity.md for the full integrity protocol.

Step 4: Route Based on Verdict

no — Claim not supported

1. Record postmortem in findings.md (Research Findings section):
   • What was tested, what failed, hypotheses for why
   • Constraints for future attempts (what NOT to try again)
2. Update CLAUDE.md Pipeline Status
3. Decide whether to pivot to next idea from IDEA_CANDIDATES.md or try an alternative approach

partial — Claim partially supported

1. Update the working claim to reflect what IS supported
2. Record the gap in findings.md
3. Design and run supplementary experiments to fill evidence gaps
4. Re-run result-to-claim after supplementary experiments complete
5. Multiple rounds of partial on the same claim → record analysis in findings.md, consider whether to narrow the claim scope or switch ideas

yes — Claim supported

1. Record confirmed claim in project notes
2. If ablation studies are incomplete → trigger /ablation-planner
3. If all evidence is in → ready for paper writing

Step 5: Update Research Wiki (if active)

Skip this step entirely if research-wiki/ does not exist.

If research-wiki/ exists, resolve $WIKI_SCRIPT per the canonical chain documented in shared-references/wiki-helper-resolution.md (Variant B — warn-and-skip for caller skills). The verdict / claim status / idea-outcome page edits below run on raw markdown and don't need the helper, but edges, query-pack rebuild, and the log line do.

cd "$(git rev-parse --show-toplevel 2>/dev/null || pwd)" || exit 1
ARIS_REPO="${ARIS_REPO:-$(awk -F'\t' '$1=="repo_root"{print $2; exit}' .debuffer_skills/installed-skills.txt 2>/dev/null)}"
WIKI_SCRIPT=".debuffer_skills/tools/research_wiki.py"
[ -f "$WIKI_SCRIPT" ] || WIKI_SCRIPT="tools/research_wiki.py"
[ -f "$WIKI_SCRIPT" ] || { [ -n "${ARIS_REPO:-}" ] && WIKI_SCRIPT="$ARIS_REPO/tools/research_wiki.py"; }
[ -f "$WIKI_SCRIPT" ] || {
  echo "WARN: research_wiki.py not found; verdict will be reported but wiki edges/query-pack/log will be skipped. Fix: bash tools/install_aris.sh, export ARIS_REPO, or cp <debuffer-repo>/tools/research_wiki.py tools/." >&2
  WIKI_SCRIPT=""
}

if research-wiki/ exists:
    # 1. Create/update experiment page deterministically
    [ -n "$WIKI_SCRIPT" ] && python3 "$WIKI_SCRIPT" add_experiment research-wiki/ \
      --slug "<exp_id>" --title "<experiment title>" --idea "idea:<active_idea>" \
      --verdict "<yes|partial|no>" --confidence "<high|medium|low>" \
      --metrics "<key metrics>" --summary "<reasoning summary>" --update-on-exist

    # 2. Add claim evidence edges only. Do not edit claim status here:
    # claim status is the proof axis owned by proof-checker / add_claim.
    for each claim resolved by this verdict:
        if verdict == "yes":
            [ -n "$WIKI_SCRIPT" ] && python3 "$WIKI_SCRIPT" add_edge research-wiki/ --from "exp:<id>" --to "claim:<cid>" --type supports --evidence "<metric>"
        elif verdict == "partial":
            [ -n "$WIKI_SCRIPT" ] && python3 "$WIKI_SCRIPT" add_edge research-wiki/ --from "exp:<id>" --to "claim:<cid>" --type supports --evidence "partial"
        else:
            [ -n "$WIKI_SCRIPT" ] && python3 "$WIKI_SCRIPT" add_edge research-wiki/ --from "exp:<id>" --to "claim:<cid>" --type invalidates --evidence "<why>"

    # 3. Update idea outcome (raw markdown, helper-free)
    Update research-wiki/ideas/<idea_id>.md:
      - outcome: positive | mixed | negative
      - If negative: fill "Failure / Risk Notes" and "Lessons Learned"
      - If positive: fill "Actual Outcome" and "Reusable Components"

    # 4. Rebuild + log (only if $WIKI_SCRIPT resolved)
    [ -n "$WIKI_SCRIPT" ] && python3 "$WIKI_SCRIPT" rebuild_query_pack research-wiki/
    [ -n "$WIKI_SCRIPT" ] && python3 "$WIKI_SCRIPT" log research-wiki/ "result-to-claim: exp:<id> verdict=<verdict> for idea:<idea_id>"

    # 5. Re-ideation suggestion
    Count failed/partial ideas since last /idea-creator run.
    If >= 3: print "💡 3+ ideas tested since last ideation. Consider re-running /idea-creator — the wiki now knows what doesn't work."

Rules

• Codex is the judge, not CC. CC collects evidence and routes; Codex evaluates. This prevents post-hoc rationalization.
• Do not inflate claims beyond what the data supports. If Codex says "partial", do not round up to "yes".
• A single positive result on one dataset does not support a general claim. Be honest about scope.
• If confidence is low, treat the judgment as inconclusive and add experiments rather than committing to a claim.
• If Codex MCP is unavailable (call fails), CC makes its own judgment and marks it [pending Codex review] — do not block the pipeline.
• Always record the verdict and reasoning in findings.md, regardless of outcome.

Review Tracing

After each mcp__codex__codex or mcp__codex__codex-reply reviewer call, save the trace following shared-references/review-tracing.md (Policy C — forensic; never silently skip). Use save_trace.sh (resolved per the chain in shared-references/integration-contract.md §2) or write files directly to .debuffer_skills/traces/<skill>/<date>_run<NN>/. Respect the --- trace: parameter (default: full).