uai-reproducibility

name	uai-reproducibility
description	Use when hardening reproducibility evidence for a UAI paper, including seeds, sampler convergence diagnostics, ELBO and calibration traces, dataset and hyperparameter disclosure, compute reporting, and honest code-availability statements, since UAI strongly encourages released code and data and reviews whether claims are convincingly backed.

UAI Reproducibility

UAI's 2026 CFP did not impose a formal reproducibility checklist (one may appear later — 待核实 each cycle), but it strongly encouraged code and data availability and instructed reviewers to judge whether claims are backed up convincingly. At this venue "convincing" has a technical meaning: stochastic-inference results carry diagnostics, not just point estimates. This skill turns that norm into an audit.

The UAI-specific bar

Reproducibility questions at a probabilistic-inference venue go one level deeper than "can I rerun the script":

• A sampler that reproduces the same posterior mean but different tail quantiles has not reproduced the paper — report and check the diagnostics that detect this (R-hat, ESS, divergent transitions where relevant).
• A variational result is reproducible when the ELBO trajectory and the selected restart are recoverable, not merely the final metric; multi-restart selection rules must be stated.
• Calibration claims reproduce only if the binning scheme, the split used for recalibration, and the α grid are all pinned down; empirical coverage moves with all three.
• Causal-discovery results depend on graph generation as much as on the algorithm: publish the SCM sampler, noise families, and intervention protocol.

Determinism ledger

Record every randomness source once, in code, and cite it from the paper:

# repro/ledger.py — imported by every experiment entry point
import json, platform, random, numpy as np

def fix_and_log(seed: int, path: str = "run_manifest.json"):
    random.seed(seed)
    np.random.seed(seed)
    manifest = {
        "seed": seed,
        "python": platform.python_version(),
        "numpy": np.__version__,
        "chains": 4, "warmup": 1000, "draws": 2000,   # sampler config lives here
        "elbo_restarts": 10, "restart_rule": "best final ELBO",
    }
    json.dump(manifest, open(path, "w"), indent=2)
    return manifest

The manifest style matters more than the specific fields: one machine-readable file per run, checked into the artifact, lets a reviewer reconcile the paper's Table 3 with an actual execution.

Disclosure map

What must be recoverable	Where it lives at UAI	Common omission
Model and assumption set	Main part, stated with each theorem/method	Assumptions distributed across three sections
Sampler / optimizer settings	Appendix (unlimited, same PDF)	"Default settings" without library version
Hyperparameter search space and selection rule	Appendix table	Only the winning configuration reported
Seeds and number of repeats	Appendix + artifact manifest	Single-run results with no variance
Dataset versions, splits, preprocessing	Appendix + loader script in ZIP	Preprocessing "as in [12]" where [12] is ambiguous
Compute (hardware, runtime, memory)	Appendix	Runtime reported only for the proposed method, not baselines
Code/data availability statement	Main part or appendix	Silence, which reviewers read as "unavailable"

Diagnostic quick reference

What "reported convincingly" tends to mean per inference family — as conventions of the field, not venue mandates:

Family	Minimum reported	Stronger version
MCMC	R-hat per parameter block, ESS, chain count/length	Rank plots; comparison against a long-run gold standard
Variational	Final ELBO, restart count and rule	ELBO traces; posterior-quality check on a tractable case
SMC / particle	Particle count, resampling scheme, ESS trajectory	Variance of the marginal-likelihood estimate over repeats
Conformal / intervals	Split sizes, α grid, empirical coverage	Conditional coverage slices; width distribution
Calibration	Binning scheme, ECE definition used	Reliability diagrams with confidence bands over seeds
Causal discovery	Graph generator, noise family, SHD/SID per seed	Sensitivity to assumption violations (unfaithfulness, confounding)

If a row's "minimum" column is missing for your method family, expect the backing criterion to absorb the damage.

Honesty over completeness

• If code cannot be released (industrial constraints, licensed data), say so in the paper and compensate: fuller pseudocode, exact hyperparameters, synthetic surrogates for private datasets. The encouraged-not-mandatory wording gives room for honesty, not for vagueness.
• Report failure modes you observed — initializations that collapse, chains that need longer warmup on one dataset. Probabilistic-ML reviewers trust papers that know where their method breaks.
• Never let variance disappear in the retelling: if three of ten seeds underperform, the aggregate table must reflect it (mean ± sd over all ten, or a stated, principled selection rule).

Where reproducibility evidence lives

Split by tier deliberately: the availability statement and diagnostic summaries in the reviewed PDF (body or appendix), where they count toward backing; manifests, loaders, and per-run logs in the ZIP, where they support spot-checks. Never leave the only mention of seeds or repeat counts inside the optional archive — reviewers grade what the PDF says.

Pre-submission reproducibility drill

1. Clone your own artifact onto a machine that never ran the project; follow only the README.
2. Reproduce the smallest headline number end to end, including the diagnostic that validates it.
3. Diff the regenerated figure against the paper's; investigate any visible deviation before a reviewer does.
4. Grep the paper for every "we observe/we find/consistently" and confirm each maps to a logged run.
5. Write the availability statement last, describing what is actually in the ZIP — not what was planned in January.

Output format

[Repro grade] turnkey / recoverable with effort / under-specified
[Diagnostics reported] <R-hat/ESS/ELBO/coverage/SHD... as applicable>
[Determinism ledger] present / partial / absent
[Disclosure gaps] <items from the map still missing>
[Availability statement] drafted / needs honesty pass / missing