ab-analysis-kit

Interactively tune an experiment's compute method in the browser cockpit: run `abk run` to persist results, open `abk explore --select <exp>`, and guide the user to turn the method's knobs (test type, alpha, CUPED, correction, bootstrap iters) on their **real** cumulative series and watch the effect + CI stabilization recompute live — with the **A/A calibration chip always visible** — then **Apply** the chosen `method_params` back into the experiment YAML in place (validated, prior version archived). Use when the user wants to tune or dial in a method by hand / interactively / in the browser, explore an experiment's results live, compare methods on real data, adjust alpha or turn CUPED on, or be in the loop while deciding a method. This is the primary, hands-on entry point for choosing a compute method.

Scaffold a new reusable abkit metric as a validated YAML file under metrics/. Use when the user wants to add or create an abkit metric, define a conversion / revenue / ratio measure for experiments, or set up a metric SQL query. Produces a metrics/<name>.yml plus one-row-per-unit SQL that joins the assignment cohort macro, passes config-lint, and is ready to reference from an experiment.

Size an A/B experiment before launch with `abk plan`: required sample size, achievable minimum detectable effect (MDE), and achieved power at the effective two-tier alpha. Use when the user asks how many users / how big a sample they need, how long to run an experiment, whether it is powered, what effect they can detect at the current size, or to compute power / MDE for a planned or running test. Read-only — never runs the pipeline or writes results. For a greenfield experiment with no data yet, supply baseline moments with `--baseline`.

Configure an abkit project's database connection in profiles.yml so `abk run` works end to end. Use for first-time setup, right after `abk init`, when the user asks to connect abkit to their database or set up profiles.yml, or when a run fails with "internal_database must be set", "data_database must be set", "Profiles file not found", "default_profile '<x>' not found", or "Connection refused". Interactive and database-type-aware.

Check whether an abkit compute method is actually calibrated on the experiment's own data by running placebo A/A splits — does its false-positive rate really equal α? Use when the user asks "is this method trustworthy on my data", "run an A/A", worries about false positives, peeking / optional stopping, whether a verdict can be believed, or before shipping a metric that a p-value looks too good on. Runs `abk validate`, reads the FPR (single-look AND peeking) / power / coverage matrix, acts on the recommendation and the budget bands, persists `_ab_aa_runs`, and lights the explore calibration chip. This is A/A CALIBRATION, not a config lint.

Report an abkit bug, request a feature, or send feedback to the maintainers as a GitHub issue on the upstream repo (alexeiveselov92/ab-analysis-kit). Use when an `abk` command errors or behaves unexpectedly and it looks like an abkit defect (not the user's config), when the user wishes abkit did something it doesn't, or when they ask to report/file an issue, send feedback, or contact the maintainers. Auto-collects diagnostic context, redacts all secrets, and never submits without explicit confirmation.

Scaffold a new abkit experiment as a validated YAML file under experiments/. Use when the user wants to add or create an A/B experiment, analyze a test or rollout, set up an experiment analysis, or compare variants on one or more metrics. Produces an experiments/<name>.yml that references existing metrics, passes config validation, and is ready to run with `abk run`.