ab-test-runner

name	ab-test-runner
description	Run A/B tests comparing convention variants to measure compliance improvements
allowed-tools	Bash, Read, Write, Glob, Grep

A/B Test Runner for Convention Iterations

Run blind A/B tests that compare two versions of a convention (e.g., beads gate wording, permission rule phrasing) and produce a scored comparison report.

When to Use

• After drafting a revised convention, before deploying it enterprise-wide
• To validate that a CLAUDE.md rewrite actually improves agent compliance
• To compare different instruction styles (imperative vs. descriptive, gate-first vs. buried)

Workflow

1. Locate or Create a Test Definition

Test definitions live in evals/ab-testing/tests/ as YAML files.

ls /root/projects/COO/evals/ab-testing/tests/

If no test exists for your convention change, create one following the schema at evals/ab-testing/ab-test-schema.yaml. Required fields:

• name: kebab-case identifier
• variant_a / variant_b: each with label and content_file (path relative to evals/ab-testing/)
• test_prompts: list of user prompts that exercise the convention
• judge_criteria: markdown describing how to score responses

Place variant content files in evals/ab-testing/variants/.

2. Run the Test

cd /root/projects/COO/evals/ab-testing
bash runner.sh tests/<test-name>.yaml

Options:

• --dry-run : Validate parsing without invoking Claude (fast sanity check)
• --report-dir <dir> : Override default report output directory

3. Review the Report

Reports are written to evals/ab-testing/reports/ as JSON:

cat /root/projects/COO/evals/ab-testing/reports/<test-name>-<timestamp>.json | jq '.summary'

The report contains:

• Per-trial scores from the blind judge (gate_recognition, correct_action, bead_id_reference, autonomy)
• Per-trial winner (A, B, or tie)
• Aggregate summary with overall verdict

4. Interpret Results

Verdict	Meaning	Action
variant_b	Revision improves compliance	Deploy variant B to conventions and zgent repos
variant_a	Current wording is better	Revise the revision or keep current
inconclusive	No clear winner	Add more test prompts, increase min_trials, or refine judge criteria

5. Writing Good Test Prompts

Include prompts across difficulty levels:

• clear_substantive: Work that obviously needs a bead (new features, refactors, dependency installs)
• clear_exempt: Work that obviously does NOT need a bead (questions, read-only diagnostics)
• ambiguous_minor: Edge cases where the convention's guidance matters most (typo fixes, config tweaks)

The ambiguous prompts are the most diagnostic -- they reveal whether the convention wording gives the agent enough guidance to make the right call.

6. Writing Good Judge Criteria

The judge is a separate Claude context with no access to the variant labels. It sees responses as "X" and "Y" (randomized per trial when blind: true).

Good criteria:

• Enumerate specific observable behaviors to score
• Use a numeric scale (1-5) for each dimension
• Include a "correct action" dimension tied to expected behavior
• Ask for a winner and justification

Available Test: beads-gate-wording-v2

The first concrete test compares Gen 2 (rule-in-body, no imperative gate) vs Gen 3 (STOP gate with bd commands at top of CLAUDE.md). This tests the core finding from the claude-monitor incident: that cross-reference gates get ignored by cold-start instances.

# Dry run to verify parsing
bash /root/projects/COO/evals/ab-testing/runner.sh \
  /root/projects/COO/evals/ab-testing/tests/beads-gate-wording-v2.yaml --dry-run

# Full run
bash /root/projects/COO/evals/ab-testing/runner.sh \
  /root/projects/COO/evals/ab-testing/tests/beads-gate-wording-v2.yaml

Architecture Notes

The runner uses three separate Claude contexts per trial:

1. Variant A agent: Given variant A's CLAUDE.md, asked to plan a response to the prompt
2. Variant B agent: Given variant B's CLAUDE.md, asked to plan a response to the prompt
3. Blind judge: Given both responses (as X/Y, randomized), scores against criteria

This three-context design ensures:

• Neither variant agent sees the other's response
• The judge cannot tell which response came from which variant
• The judge criteria are consistent across all trials