| name | computational-experiment-rigor |
| description | Rigor protocol for the analysis itself — uncertainty quantification, convergence checks, classification thresholds, literature grounding, and reproducibility records. Use when designing or running a simulation, numerical experiment, or parameter study, when computing estimates (exponents, rates, fits, spectra, regime classifications), or before making any quantitative claim — i.e. while producing the numbers, not while writing them up. |
Computational Experiment Rigor
When to use
Read this skill before finalizing any run that estimates numbers from simulation
or data (exponents, rates, fits, spectra, regime classifications, benchmark
metrics). Apply it proportionally: a quick demo plot needs only items 1 and 6;
a regime classification or quantitative claim needs all of them.
Protocol
1. Recognize canonical systems and ground in known results
If the system under study is a standard one (e.g. driven pendulum, Lorenz,
logistic map, Ising, double pendulum, Duffing, Hodgkin–Huxley), say so by name,
state the commonly reported behavior for the studied parameter values from
literature you are confident about, and compare your results against it.
- When your result disagrees with the commonly reported one, do not silently
report yours: flag the discrepancy and investigate the usual causes —
initial conditions and multistability (coexisting attractors), transient
length, integrator step, finite observation time.
- Name sources at the textbook/author level only when confident (e.g.
"Baker & Gollub, Chaotic Dynamics"). Never fabricate precise citations,
page numbers, DOIs, or numeric reference values you are unsure of; prefer
"commonly reported as ..." with an explicit confidence statement.
2. Quantify uncertainty on every decision-relevant estimate
A point estimate near a decision boundary is not a result.
- Repeat estimates across ≥3 random seeds and ≥2 observation durations for
every quantity that drives a classification or conclusion — not just one
showcase case.
- Report mean ± sample standard deviation (or min–max range) in the table and
CSV, not bare point values.
- Classification threshold rule: if |estimate| < 3× its spread, the value is
indistinguishable from zero — label it "marginal / not resolved at this
precision" and either extend the run or use an independent discriminator.
Do not assign a definitive class from an unresolved estimate.
3. Convergence and validity checks
- Verify at least one representative case per regime class against a halved
timestep and a doubled duration; state the relative change.
- For Hamiltonian-leaning or long integrations with fixed-step methods, check
a conserved/slowly-varying quantity (energy drift, phase-space volume) and
report it.
- State integrator, step size, transient discarded, and observation window in
the report.
4. Triangulate classifications with independent discriminators
Never classify dynamical regimes from a single statistic. Cheap corroborations
you usually already have:
- Strobe/Poincaré point counting → identifies period-n orbits explicitly
("3 distinct strobe points → period-3"), distinguishes quasiperiodic
(closed curve) from chaotic (folded scatter).
- Power spectrum of a long sample → discrete lines vs broadband.
- Sensitivity probe → rerun from a 1e-8-perturbed initial condition and
report divergence or convergence of trajectories.
State which discriminators agree and which were not run.
5. Reproducibility record
The report and code must let a colleague regenerate every number: record
seeds, initial conditions, all parameter values, step sizes, durations, and
the exact commands run. Idempotent script entrypoints beat notebook-style
ad-hoc cells.
6. Honest accounting
- Report wall-clock time and compute time separately and label them; never
imply the whole study took only the inner-loop compute time.
- Failed or dead-end attempts that shaped the method belong in one short
"what didn't work" line, not hidden.
- Every conclusion gets a confidence level (high/medium/low) tied to the
evidence above.
Delegation note
When a verification subtask is delegated to a subagent, give it this skill's
standards explicitly in the task description (seeds, durations, threshold
rule), and reconcile by comparing its numbers to yours with the spread, not
just "consistent".
Results contract (Intelligence: Pro)
On Pro-intelligence runs the harness enforces a results contract on the final
answer (mean ± spread on every decision-relevant estimate, the decision rule
stated and applied per row, a classification table, projection-aliasing noted,
basin/IC-dependence for borderline cases, and a Limitations paragraph) and sends
the run back for revision when any is missing. The authoritative wording is
injected into the system prompt on those runs; follow it verbatim and meet it on
the first pass to avoid a revision round-trip. The items above (UQ, the 3×-spread
rule, triangulation, limitations) already satisfy it.
