experiment

name	experiment
description	Promote branch-tagged traces into a durable Opik dataset + experiment with a UI scorecard. Outer loop of the eval pipeline. Use when the score must outlive the branch — cross-time comparison, reviewer handoff. Invoke as `edd:experiment` or when the user says "build dataset", "run experiment", "scoreboard in Opik UI", "outer loop".

edd:experiment — dataset → experiment → inspect

Mode 2 Phase 2 of the eval pipeline. Promotes the inner-loop score into a durable artifact.

Preconditions

• .edd/session.json has project and branch_tag (<git-branch>)
• .edd/evaluator-plan.md exists (edd:scope-evals)
• Smoke-judge-check passed in edd:run — every judge in the plan landed at least one score on the 3–5 trace sample. This is a harness probe, not a green score-table (Mode 2 doesn't have one). If anything is silent, fix it before scaling.
• Sim batch tagged in the last few hours (re-run edd:run if stale)
• venv active

When to invoke

• comparing two prompt variants on the same inputs
• iterating across days and needing a persistent scorecard
• handing the score off to a reviewer who wasn't in the inner loop

Phase A — Pick the evaluator

Read .edd/evaluator-plan.md. Pick the single dimension that the diff most directly exercises (you can add more evaluators to the experiment run later — start with one). Consult references/evaluator-selection.md if uncertain.

The chosen evaluator's schema name (not id) flows through every later command.

Phase B — Design the dataset

Follow references/dataset-design.md. Decide:

• naming per conventions — read project + topic from session, start at v1, bump when the item shape changes
• coverage mix — every promise represented; aggression-1 + 2 mix unless the experiment specifically targets adversarial
• whether metadata propagates through the trace (rare — usually trace input/output is enough)

Write dataset_name to .edd/session.json.

Phase C — Tag a sim batch

If the last edd:run was less than ~6h ago you can reuse those traces. Otherwise re-run edd:run with full enrichment — judges in the experiment read the same paths they used in the inner loop.

Phase D — Build the dataset

edd-build \
  --project <opik-project> \
  --dataset-name <dataset_name> \
  --description "<one-line summary — topic + hypothesis>" \
  --branch-tag $(git rev-parse --abbrev-ref HEAD) \
  --from "$(date -u -v-6H +%Y-%m-%dT%H:%M:%SZ)" \
  [--tag <extra-tag>] \
  [--extractor _local.my_extractor:extract] \
  --dry-run

--description is required (Opik dataset card uses it). --dry-run first — verify item count and shape, then re-run without it.

Default extractor reads metadata.user_message + metadata.assistant_response. Supply --extractor module:function if your runtime emits trace paths that differ.

Tags auto-applied: --branch-tag + any --tag you pass. Branch name = topic = tag — <topic> is the single identity across traces, dataset, and experiment. Mode and aggression go in --description. Branch-tag warning prints if it references main/master — pass --allow-main to silence.

Phase D.5 — Expand for coverage (gate before Phase E)

Hard ordering rule: if you're going to grow the dataset with AI variants, do it here, between build and judge. Running Phase E on a thin seed and then expanding wastes judge spend on a stale scorecard you can't compare against the expanded one.

Stream a sample of the seed items and compare actual coverage against the target mix in references/dataset-design.md. If any bucket is materially under-represented (happy-path, wording variants, edge cases, adversarial, multi-turn, negatives), hand off to edd:expand. The expand skill derives variation_instructions and preserve_fields from the actual gap; it never sits between two judge runs.

If the seed already covers the target mix at the size you need, skip expand and go straight to Phase E.

Phase E — Run the experiment

edd-run \
  --project <opik-project> \
  --dataset-name <dataset_name> \
  --evaluator "<schema-name-a>,<schema-name-b>" \
  --description "<hypothesis — what changed and what we expect>" \
  --branch-tag $(git rev-parse --abbrev-ref HEAD) \
  [--tag <extra-tag>] \
  [--score-timeout 300] \
  [--dry-run]

--description is required and surfaces on the Opik experiment card.

This:

1. Triggers each evaluator on every linked trace
2. Polls scores until done or timeout
3. Creates the experiment with model + branch + evaluator metadata
4. Copies scores onto experiment items

Write experiment_name to .edd/session.json.

Phase F — Inspect + iterate

edd-inspect \
  --experiment-name <experiment_name> \
  [--score-threshold 0.5] \
  [--evaluator "<Schema Name>"] \
  [--out-jsonl /tmp/exp.jsonl]

Prints a per-evaluator digest + every failure below threshold with trace links.

Classify failures via references/failure-modes.md:

Class	Action
prompt issue	fix prompt/skill/tool, re-run Phase E against the same dataset for a fresh experiment
dataset issue	patch scenarios, rebuild dataset (Phase D), new version — bump v<N>
evaluator issue	back to edd:scope-evals, recalibrate, re-run Phase E
flaky / model-bound	tag and skip

Apply the stopping rules — two prompt iterations on the same red judge ⇒ widen the search.

Anti-patterns

• Running an experiment without an evaluator — "I want a scoreboard" is not an evaluator; pick the dimension first.
• Skipping --dry-run — bad extractor + real write = polluted dataset.

See also: pipeline anti-patterns (global rules, including dataset rewrite-in-place).

Next

• Inspect digest surfaces persistent failures on one dimension → fix the prompt / dataset / evaluator and re-run from Phase D with a fresh dataset version (<topic>-v<N+1>). Expansion happens before judges, so a new comparison run means a new build.
• Persistent failures suggest the coverage mix is wrong → bump the dataset version, return to Phase D.5 to re-design coverage with edd:expand, then judge.