test-napkin-math

name	test-napkin-math
description	Use after any change under experiments/napkin_math/ or to the upstream skill prompts that feed into it (extract-parameters-from-full, extract-parameters-from-digest, generate-bounds, generate-calculations, run-scenarios, monte-carlo). Runs the smoke-test suite and reports pass/fail. Invoke before declaring napkin-math work done.

Test the napkin_math experiment

Overview

A single-shot smoke check for the napkin_math experiment. The test logic lives in experiments/napkin_math/tests/run_smoke.py; this skill is a thin wrapper that invokes the script, parses its output, and reports a one-line summary.

Covers the Monte Carlo runner (end-to-end, determinism, Bernoulli arithmetic, sensitivity ranking), the strict-schema fail-fast paths (each required field individually), prepare_extract_input.py import sanity, and the compress_report_section pytest suite.

When to Use

• After any edit to:
  • experiments/napkin_math/run_monte_carlo.py
  • experiments/napkin_math/prepare_extract_input.py
  • experiments/napkin_math/tests/fixtures/smoke/*
  • worker_plan/worker_plan_internal/parameter_extraction/compress_report_section.py
  • any system-prompt.txt / SKILL.md under experiments/napkin_math/.claude/skills/{extract-parameters-from-full,extract-parameters-from-digest,generate-bounds,generate-calculations,run-scenarios,monte-carlo}/ that touches the artifact schema
• Before declaring any napkin_math change "done" — even if the change looks self-contained, the schema is tightly coupled across stages
• When the user asks "did I break anything?" or "run the napkin_math tests"

Not for: end-to-end LLM-driven runs of the extract-parameters-from-full skills against real reports (those require an LLM in the loop and are out of scope here).

Workflow

1. Invoke the runner. Requires Python 3.11+ with NumPy and pytest installed:

   /opt/homebrew/bin/python3.11 experiments/napkin_math/tests/run_smoke.py
   

   Override the interpreter with NAPKIN_TEST_PYTHON=<path> if the default isn't available.

2. Read the script's stdout. It prints one section per check with individual ok / FAIL lines, then a final summary:

   SUMMARY: 7/7 checks passed
   ALL GREEN
   

   Exit code is 0 on full pass, 1 on any failure.

3. Report back to the user. On success: one line ("All 7 napkin_math smoke checks passed."). On failure: list the failing checks with the detail printed by the runner, and point at the specific file or schema field implicated.

What the seven checks cover

Check	What it verifies
end_to_end	Runner consumes the synthetic fixture, emits both expected outputs, zero warnings
determinism	Two runs with the same seed produce byte-identical JSON
bernoulli_arithmetic	Mean of total_budget_with_gate_inr ≈ 1,000,000 + 0.6 * 250,000 within ±5,000
sensitivity_ranking	Bernoulli gate is the sole driver of its dependent output (correlation == 1.0); two-input convert sensitivity has both inputs
schema_errors	Each required schema field (sampling_discipline, non_negative, default_pass_probability for bernoulli_gate, output_name, output_unit) triggers a SCHEMA ERROR exit code 2 with a message naming the upstream stage to re-run
prepare_extract_input_imports	prepare_extract_input.py imports cleanly and exposes build_combined_digest
compress_pytest	The pytest suite for compress_report_section.py passes (13 tests)

Common Mistakes

Mistake	Fix
Skipping the check because "the diff is tiny"	The schema is tightly coupled across five skills + the runner. A tiny change in one prompt can break the runner's strict validation. Always run it.
Reporting "tests pass" without running the script	If you didn't see ALL GREEN, you didn't pass.
Running with python3 instead of python3.11	NumPy and the worker_plan tests live in the 3.11 env. Use the explicit path or set NAPKIN_TEST_PYTHON.
Treating a compress_pytest failure as unrelated	The compressor is part of the napkin_math experiment; if its tests break, the digest produced by prepare_extract_input.py is suspect.
Editing fixtures to make tests pass	Fixtures encode the contract. If a fixture needs to change, the schema or the runner changed for a real reason — update both consciously and re-justify each touched check.

Reference

• Test runner: experiments/napkin_math/tests/run_smoke.py
• Synthetic fixture: experiments/napkin_math/tests/fixtures/smoke/
• Monte Carlo runner under test: experiments/napkin_math/run_monte_carlo.py
• Companion skills (consumers of the same schema): ../extract-parameters-from-full/SKILL.md, ../extract-parameters-from-digest/SKILL.md, ../generate-bounds/SKILL.md, ../generate-calculations/SKILL.md, ../run-scenarios/SKILL.md, ../monte-carlo/SKILL.md