braintrust-attribute-multi-variable-change

name

braintrust-attribute-multi-variable-change

description

Attribute an observed change when several things moved at once — model plus prompt plus tools, a provider migration, a framework upgrade, or a vendor swap that bundles serving stack with model. Use when asked which part of a change caused the result, when a comparison's arms differ in more than one way, when a treatment has no uniform implementation across vendors, or when a serving-stack difference is confounded with a model difference. Do not use for a clean single-variable comparison, or to design an experiment that has not yet run.

Attribute a multi-variable change

Contract: references/interaction-contract.md. Calibration, templates, provenance: references/isolation-designs.md.

Trigger

• "We upgraded the model and rewrote the prompt — which one helped?"
• A vendor or provider migration bundling model, serving stack, and tool surface.
• Arms that differ in more than one respect, discovered after results exist.

Do

1. Enumerate what actually differs between arms with the inventory in reference.md, including the variables hiding inside apparently atomic ones: the serving path behind a model string, the tool manifest behind "the agent."
2. State honestly what the existing data can and cannot separate. Two arms differing in three ways support one claim about the bundle and no claim about any component. Say that before analyzing anything.
3. Choose the cheapest design that isolates what the decision needs — ablation ladder, one-factor-at-a-time from the new baseline, or full factorial only when interactions are the actual question.
4. Where a factor cannot be held constant — a vendor with no equivalent parameter — record the per-arm implementation as a published table and exclude that arm from the component claim rather than scoring it "no benefit."
5. Report bundle effect and component effects as , each with its own uncertainty.