uai-experiments

name	uai-experiments
description	Use when designing or auditing UAI experiments where inference quality is the endpoint, covering calibration and coverage measurement, posterior diagnostics, structure-recovery metrics for causal and graphical models, seeded repeated runs, baselines from both sampling and optimization families, and mapping each claim to its evidence.

UAI Experiments

Use this while the empirical design is still changeable. At UAI the object under test is usually an inference procedure — a posterior, a graph, an interval, a decision policy — so the experimental question is rarely "is accuracy higher?" and usually "is the uncertainty right, and at what cost?". Reviewers score whether claims are backed up convincingly; design the study so each claim has a designated exhibit.

Metrics follow the probabilistic object

Claimed object	Primary metrics	Supporting diagnostics
Posterior approximation	Wasserstein/KL to gold-standard posterior on tractable cases	R-hat, ESS, trace plots; ELBO with restarts
Predictive uncertainty	NLL, CRPS, empirical coverage vs nominal	Reliability diagrams; ECE with stated binning
Conformal / interval methods	Coverage at each α, interval width	Conditional coverage slices, not just marginal
Causal structure	SHD, SID, edge precision/recall vs ground truth	Performance vs sample size; sensitivity to faithfulness violations
Treatment effects	Bias/RMSE on ATE/CATE with known ground truth	Overlap diagnostics; propensity calibration
Decision policies	Regret, expected utility under the stated prior	Robustness under prior misspecification

The recurring UAI failure is a proxy mismatch: claiming better uncertainty while measuring only accuracy, or claiming a better posterior while reporting only downstream prediction. Pick the metric that measures the claimed object directly.

The synthetic-to-real ladder

Because ground-truth posteriors and ground-truth graphs exist only where you construct them, strong UAI papers climb a ladder:

1. Exact-truth regime: small models where exact inference or the true SCM is available — the only place "closer to the truth" is literally checkable.
2. Stress regime: the same generators with an assumption deliberately broken (unfaithful distributions, heavy-tailed noise, hidden confounding) — this is where your limitations section gets its numbers.
3. Real-data regime: demonstrates relevance, evaluated by proxies (held-out NLL, stability, downstream decisions) with the proxy status stated plainly.

A paper living only on rung 3 cannot back an inference-quality claim; one living only on rung 1 will be asked why anyone should care. Budget experiments across all three.

Stochasticity is part of the result

• Repeat over seeds and report dispersion (mean ± sd, or paired comparisons when methods share data draws); single-run tables at an uncertainty venue are self-refuting.
• Distinguish the two randomness sources — data generation and algorithm internals — and vary them separately when the claim depends on one of them.
• Fix the comparison protocol before running: same data splits, same compute budget or an explicit cost axis, same tuning effort for baselines as for the proposed method.

# Paired, seeded comparison harness: every method sees identical data draws
import numpy as np

def run_grid(methods: dict, make_data, seeds=range(10)):
    rows = []
    for s in seeds:
        data = make_data(rng=np.random.default_rng(s))   # shared draw per seed
        for name, fit in methods.items():
            post = fit(data, seed=s)
            rows.append({"seed": s, "method": name,
                         "coverage@90": post.coverage(0.90),
                         "nll": post.nll(data.test),
                         "ess_min": post.min_ess()})
    return rows   # aggregate as mean ± sd; report per-seed table in the appendix

Baseline selection that survives this reviewer pool

• Include the cheap classical baseline (exact inference where feasible, a well-tuned Laplace approximation, the PC algorithm, plain split conformal): UAI reviewers use these as sanity anchors, and their absence reads as fear.
• Include the strongest recent neighbor from UAI/AISTATS/NeurIPS, tuned in good faith with its search grid disclosed.
• When your method has a compute knob (chains, particles, restarts), plot the quality-versus-cost curve rather than a single operating point chosen post hoc.

Compute fairness, concretely

• Match tuning budgets: if the proposed method saw 50 configurations, baselines see 50, drawn from grids their authors would endorse.
• Match convergence criteria: comparing your converged sampler against a baseline's fixed-iteration run measures patience, not methods.
• When budgets cannot match (an exact baseline that only scales to n=100), report it at its feasible scale and mark the cell honestly rather than dropping the method.
• State who tuned what: "baseline hyperparameters from the original paper" and "tuned by us on the same validation split" are different evidentiary claims.

Reporting block, standardized

Give every experiment family the same reporting block in the appendix, so reviewers can audit uniformly and you can spot your own gaps:

EXPERIMENT <id> — backs claim: <paper sentence, quoted>
  data:        <generator or dataset+version, splits, preprocessing>
  methods:     <proposed + baselines, tuning grids, selection rule>
  randomness:  <seeds, what varies per seed: data draw / init / both>
  compute:     <hardware, wall-clock per method>
  metrics:     <primary + diagnostics, with definitions or citations>
  result:      <table/figure reference; dispersion form (sd / CI / paired)>
  caveats:     <regimes where the result did not hold>

The caveats line is not decoration. At this venue an experiment section that admits where the method loses reads as calibrated; one that never loses reads as curated.

Ablations for mechanisms, not rituals

Design each ablation to isolate the component your theory says matters: remove the coupling, swap the score function, freeze the calibration step. An ablation grid nobody can interpret is appendix filler; a single ablation matching a theorem's prediction is evidence.

Output format

[Claim → exhibit map] <each headline claim with its table/figure/diagnostic>
[Ladder coverage] exact-truth / stress / real — which rungs are missing
[Uncertainty of results] seeds, dispersion, pairing — adequate? 
[Baseline audit] classical anchor present? strongest neighbor tuned fairly?
[Proxy mismatches] <claims measured by the wrong metric>