| name | failure-checklist |
| description | AI Research 7 大 known failure mode checklist：data snoop / distribution mismatch / leakage / weak baseline / single-seed / pre-trained sanity / implementation diff。Stage 推进前必须逐项 ✓。Use when user says "failure checklist", "7 种坑", "data snoop", "leakage check", "sanity check", "before submission check". |
| argument-hint | [--run-id <id>] [--interactive] |
| allowed-tools | Bash(*), Read, Write |
/failure-checklist — A5 7-Mode Blocking Gate
借鉴 ARS Failure-Mode Checklist + design.md Part V P1 A5。
Overview
7 项逐项确认（agent + user 联手）：
| # | Failure mode | 怎么 verify |
|---|
| 1 | Data snoop | eval set 没在训练中用过（grep dataset split） |
| 2 | Distribution mismatch | train/eval domain 一致（mean/std 列表对比） |
| 3 | Leakage | features 不含 label 信息（如 timestamp leak） |
| 4 | Weak baseline | /run-zero 用的是 paper 原 config（git diff configs/paper.yaml 空） |
| 5 | Single-seed | ≥3 seed mean ± std |
| 6 | Pre-trained sanity | 未 fine-tune ckpt SR < 5%（防 "其实是 zero-shot ability"） |
| 7 | Implementation diff | commit clean / /vla-audit 通过 |
Workflow
RUN=${RUN:-$(ls runs | tail -1)}
INTERACTIVE=${INTERACTIVE:-true}
for i in 1 2 3 4 5 6 7; do
result=$(python3 tools/failure_check.py $i --run $RUN)
if [ "$result" != "PASS" ] && [ "$INTERACTIVE" = "true" ]; then
AskUserQuestion "$i: $result. mark anyway? [Y/n]"
fi
done
Output
.auto-production/audit/a5_failure_checklist.passed.auto-production/audit/a5_failure_report.json
Composition
- RESULTS gate + FINAL gate 必检
/pivot 决策 PROCEED 前必跑
