bayesian-methodology

name	bayesian-methodology
description	Use when designing, reviewing, or changing Bayesian confidence, calibration, Beta-Binomial updating, prior specification, posterior computation, Thompson/LCB ranking, Brier/ECE calibration, alpha/beta parameters, uncertainty-aware scoring, or probabilistic feedback loops. Also use when a system proposes hardcoded confidence values, magic thresholds, or point estimates where distributions should carry the evidence.
compatibility	Reference doctrine reads with no tooling. Bundled scripts require Python 3.8+ and the standard library only.
metadata	{"self_test":"python3 scripts/self_test.py"}

Bayesian Methodology

Use Bayesian machinery as machinery, not vocabulary. A confidence score that cannot be updated by evidence is not Bayesian. A posterior that is evaluated on the same data that updated it is not calibrated evidence. A ranker that collapses uncertainty into one unexplained constant has already thrown away the useful part of the model.

Read references/failure-modes.md first when debugging or extending a confidence system. Then use the sections below to check the math and the engineering contract.

Core Mechanics

Beta-Binomial Update

For a Bernoulli success probability theta:

theta ~ Beta(alpha, beta)
observe s successes and f failures
posterior = Beta(alpha + s, beta + f)

The update is exact under the conditional-independence assumption. Add successes to alpha and failures to beta. Anything else needs an explicit likelihood model.

Effective Sample Size

For Beta(alpha, beta), prior effective sample size is:

ESS = alpha + beta

The posterior mean is the weighted average of the empirical rate and prior mean:

E[theta | data] =
  [n / (n + ESS)] * (s / n)
  + [ESS / (n + ESS)] * (alpha / ESS)

This is the central design knob. Beta(50,50) is not "neutral"; it is 100 pseudo-observations. A hardcoded confidence constant is the limiting case of an infinite-ESS prior: no future evidence can move it.

Summary Statistics

For Beta(alpha, beta):

mean     = alpha / (alpha + beta)
mode     = (alpha - 1) / (alpha + beta - 2), only when alpha > 1 and beta > 1
variance = alpha * beta / ((alpha + beta)^2 * (alpha + beta + 1))

Credible intervals and LCB/UCB scores require the Beta quantile. Do not hand-roll an inline approximation in application code; use a tested numerical routine. The bundled scripts/bayes_calc.py provides a dependency-free sanity checker for ordinary parameter ranges.

Prior Specification

Use the mean-concentration parametrization:

mu = alpha / (alpha + beta)
kappa = alpha + beta
alpha = mu * kappa
beta  = (1 - mu) * kappa

Choose mu from domain knowledge and choose kappa as how much evidence the prior should be worth. Weakly informative priors usually have small concentration, often 1-5 pseudo-observations. Beta(2,2) is a practical default when you need to avoid boundary estimates but have little domain information.

Before deploying a prior, run a prior predictive check: draw theta from the prior, simulate outcomes, and ask whether those success rates are plausible for the intended domain.

Calibration

Use Brier score as the primary scoring rule for binary probabilistic forecasts:

Brier = mean((prediction - outcome)^2)

Brier is strictly proper: truthful probabilities are optimal in expectation. ECE is useful as a diagnostic but is sensitive to bin choices, finite-sample bias, discontinuities, and gaming by uninformative predictions.

Never assess calibration on the same observations used to update alpha and beta. Use held-out data, temporal splits, or pre-update predictions evaluated against future outcomes.

Ranking Under Uncertainty

When posteriors rank candidates, choose the exploration rule deliberately.

• Thompson sampling draws one sample from each candidate posterior and ranks by the draw. It explores in proportion to posterior uncertainty.
• Lower confidence bound ranks by a pessimistic quantile, such as Beta.quantile(0.05, alpha, beta). It is safer when bad candidates are costly.
• Upper confidence bound is optimistic and should be reserved for settings where exploration cost is acceptable.

Do not tune ranking weights by taste. Track outcomes and evaluate whether ranking changes improve held-out performance.

Temporal And Hierarchical Extensions

The plain Beta-Binomial model assumes a fixed theta. If success probability changes over time, use explicit temporal adaptation. See references/temporal.md.

When candidates belong to categories with shared base rates, hierarchical models can borrow strength. See references/hierarchical.md.

Decision Procedure

Before changing confidence or calibration code:

1. Identify whether each value is a prior, likelihood, posterior, score, or decision threshold.
2. Find every hardcoded number between 0 and 1 and ask whether evidence can move it.
3. Trace the data flow from outcome observation to alpha/beta update to final score or rank.
4. Verify success increments alpha and failure increments beta under the stated model.
5. Run prior predictive checks for new priors.
6. Evaluate calibration on held-out or future data with Brier score, plus ECE only as a secondary diagnostic.
7. State what remains unverified.

Bundled Tools

Tool	Run	Purpose
scripts/bayes_calc.py	python3 scripts/bayes_calc.py <subcmd> ...	Beta-Binomial calculator: posterior, summary, decompose, quantile, forget, moments. JSON output; exit 2 on invalid input.
scripts/bayes_contract_check.py	python3 scripts/bayes_contract_check.py verify	Deterministic contract checks for common stale Bayesian claims and update arithmetic.
scripts/self_test.py	python3 scripts/self_test.py	Falsifiable regression suite for the bundled tools.

Example: 5th-percentile LCB for Beta(7,5):

python3 scripts/bayes_calc.py quantile --alpha 7 --beta 5 --p 0.05

Reference Files

File	When to read
references/failure-modes.md	Before debugging or extending confidence code
references/calibration.md	When scoring or validating probabilistic forecasts
references/temporal.md	When evidence gets stale or regimes change
references/hierarchical.md	When category-level sharing may help sparse candidates
references/ranking.md	When posteriors influence ranking or selection

Done Means

• The update rule is explicit.
• Priors have interpretable ESS and prior-predictive checks.
• Calibration uses held-out or future outcomes.
• Ranking policy is tied to exploration cost.
• Claims are backed by failable checks, not Bayesian-flavored prose.