| name | set-up-ab-testing |
| description | Use this to test an LLM change (new prompt, new model, new retrieval) on real traffic before rolling it out to everyone. Trigger on "A/B test my prompt", "roll out a new model safely", "compare two prompts in production", "canary this change", "does this actually improve things for real users". Measure impact on real users, gated, before a full switch. |
| license | CC0-1.0 |
A/B test LLM changes in production
Offline evals tell you a change should be better; a production experiment tells you it is, on real users and real traffic. For anything user-facing, run a controlled rollout instead of flipping the switch for everyone.
When to use this vs offline evals
- Offline evals (
add-llm-evals): fast, cheap, run in CI, catch regressions before deploy. Do these first, always.
- A/B / online experiment: use when the change is user-facing and the offline metric does not fully capture success (helpfulness, engagement, task completion, downstream conversion). Offline says "passes"; A/B says "users actually do better."
Set it up
- Pick one variable. New prompt OR new model OR new retrieval, not all at once, or you cannot attribute the result.
- Define the metric up front. The primary success metric (task completion, thumbs-up rate, downstream action) plus guardrail metrics (cost, latency, error rate) that must not regress. Decide the win condition before you start.
- Split traffic deterministically by user/session (a feature-flag / experiment tool: LaunchDarkly, GrowthBook, or your platform's built-in). Keep a user on one variant for consistency.
- Instrument both arms with the same tracing + online evals + feedback capture (
instrument-llm-observability, collect-user-feedback), tagged with the variant, so every metric is comparable across arms.
- Run until it is significant, not until it looks good. Small differences need enough traffic; stopping early on a favorable blip is how you ship noise.
Read the result honestly
- Compare the primary metric across arms and the guardrails (a +2% quality win that doubles cost or latency may not be worth it).
- Watch for novelty effects and segment differences (a change can help new users and hurt power users).
- If it is flat or worse, keep the control. "No worse and cheaper" is still a win.
Safer rollout patterns
- Canary: send 5% to the new variant, watch the dashboards, ramp up if healthy. Good for risky changes.
- Shadow / offline replay: run the new variant on copied traffic without showing users, compare outputs. Zero user risk, no live metric though.
Verify
- Traffic is actually split and each user stays on one arm.
- Both arms emit the same metrics, tagged by variant.
- The decision cites the primary metric + guardrails with enough sample size, not a first-hour blip.
Anti-patterns
- Changing several things at once (you cannot attribute the effect).
- Peeking and stopping the moment it looks good (false positives).
- Measuring only quality, ignoring cost/latency guardrails.
- A/B testing something an offline eval would have caught for free (do offline first).
Grounding
Online controlled experimentation methodology: Kohavi, Tang & Xu, Trustworthy Online Controlled Experiments (2020), and Kohavi et al., Controlled experiments on the web (2009). Applied here with LLM-specific metrics (online LLM-as-judge, Zheng et al. 2023, arXiv:2306.05685) and user feedback (collect-user-feedback).
