Menu
Compute and interpret MAD-based confidence scores for experiment results. Use when logging experiment results after 3+ data points to determine if improvements are real or within noise.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Based on SOC occupation classification
Use when reviewing a completed implementation holistically - a finished branch, a completed plan step, or a diff before PR. Reviews every changed line against the implementation plan (if one exists) and high engineering standards across seven dimensions, verifies tests by deliberately breaking crucial code, and emits severity-classified findings with a PASS/FAIL verdict.
Use when performing a holistic code review of an implementation - routes to codex:rescue or falls back to a direct subagent, both running the house-style:holistic-review skill
ALWAYS use before any implementation work, review, or planning: designing, writing, editing, refactoring, debugging, reviewing, or generating source code, tests, scripts, config-as-code, or behavior-affecting files. This is a baseline engineering skill and should be invoked even when a more specific language, framework, testing, or database skill also applies.
Use when asked to write, draft, file, or open a GitHub issue - mandates exploring the relevant code to understand the whole problem space, reasoning any proposed solution, and structuring the issue with summary, what/why, references, nuances, open questions, and acceptance criteria. Adds professionalism and no-sensitive-info discipline for public repos.
Use when creating, editing, triaging, or cleaning up GitHub issues - a mechanical pass that verifies the title is accurate and current, labels/assignees/milestone are correct, and the body is valid, well-formed markdown, all via the gh CLI. Does not judge content quality.
Use when running Python, installing Python packages, creating virtualenvs, or executing Python CLI tools - mandates uv/uvx over direct python/pip/pipx/venv calls when uv is installed, and gives the drop-in command mappings
| name | confidence-scoring |
| description | Compute and interpret MAD-based confidence scores for experiment results. Use when logging experiment results after 3+ data points to determine if improvements are real or within noise. |
| user-invocable | false |
Determines whether an observed improvement is real or within measurement noise using Median Absolute Deviation (MAD).
confidence: null.Given all metric values in the current segment (positive values only):
|value - median|. Take the sorted median of those absolute deviations.keep-status metric value (respecting optimization direction).|best_kept - baseline|delta / MADnull — no measurable noise to compare against.keep results exist yet: return null.null — no improvement to score.The confidence score is a multiple of the session's noise floor:
| Score | Meaning | Action |
|---|---|---|
| ≥ 2.0× | Improvement likely real | Safe to trust |
| 1.0×–2.0× | Marginal — could be noise | Consider re-running to confirm |
| < 1.0× | Within noise floor | Treat as no improvement |
When logging an experiment result to autoresearch.jsonl:
"confidence": null in the JSONL record.keep vs discard: the confidence score is advisory. It never auto-discards. But flag improvements below 1.0× in your ASI notes as "within noise — may not be real."Runs: [15200, 15400, 14800, 15100, 14600]
Median: 15100
Deviations: [100, 300, 300, 0, 500] → sorted: [0, 100, 300, 300, 500]
MAD: 300
Baseline: 15200 (first run)
Best kept: 14600
Delta: |14600 - 15200| = 600
Confidence: 600 / 300 = 2.0× ← improvement is real