auto-experiment

name	auto-experiment
description	Domain-agnostic Auto-Experiment framework for systematically improving any measurable outcome. Use when the user wants to optimize, benchmark, A/B test, or iteratively improve any metric — whether it's code performance, prompt quality, conversion rates, accuracy, latency, cost, or any numeric target. Also use when the user says "run experiments", "optimize X", "test which approach is better", "benchmark this", "systematic improvement", "A/B test", "auto-experiment", or any request to compare multiple approaches to a measurable goal. Do NOT use for one-off fixes, subjective tasks without a numeric metric, or single conversions with no comparison.
version	1.0.1
author	fft_nano
tags	["experiments","optimization","benchmarking","research","metrics"]

Auto-Experiment

Domain-agnostic Auto-Experiment orchestration using the main agent's native tools.

This is the strategy layer — how to think about, design, and run experiments. The Pi autoresearch extension is the automation layer for code optimization inside the coding runtime. Use this skill when you are in the main chat/runtime and need to orchestrate experiments across any domain.

When to Use

• A measurable metric exists (or can be defined) that might improve
• Comparing multiple approaches to the same goal
• The solution space is large and intuition is unreliable
• User phrases: "optimize", "benchmark", "A/B test", "which is better", "make it faster", "systematic improvement"

When NOT to use: One-off fixes, purely subjective tasks ("make it prettier"), single conversions with no comparison, anything lacking a numeric metric.

The 5-Phase Loop

HYPOTHESIS → BENCHMARK → EXECUTE → ANALYZE → DECIDE → repeat

Phase 1: HYPOTHESIS

• Define the metric: METRIC <name>=<value> must be possible
• Define the hypothesis: what change will improve it?
• Define the baseline: measure current state before any changes
• Rule: If you can't write a baseline METRIC line, the experiment is ill-defined. Stop and refine.

Phase 2: BENCHMARK

• Write a measurement script that outputs METRIC <name>=<numeric_value>
• Design controls: what stays identical between runs?
• Determine sample size: min 3 runs per variant, ideally 5-10
• Rule: The benchmark must be deterministic and reproducible

Phase 3: EXECUTE

• Create variants: control (baseline) + treatment(s) (your changes)
• Run in parallel via sessions_spawn when possible
• Collect raw results into structured files
• Rule: Never run only once. Noise dominates single samples.

Phase 4: ANALYZE

• Aggregate results across runs (use scripts/aggregate.py)
• Compute mean, stddev, effect size, confidence interval
• Rule: If variance > 20% of the mean, you need more samples

Phase 5: DECIDE

• Keep — significant improvement. Commit it, document it.
• Discard — noise or regression. Revert it.
• Modify — results suggest a different hypothesis. Form a new one.
• Scale — worked locally. Now test at scale or under different conditions.

Experiment Artifacts

Every experiment creates a workspace:

experiments/<name>-<date>/
├── experiment.md          # Bible: hypothesis, metric, design, results
├── benchmark.sh           # Measurement script (must output METRIC lines)
├── variants/
│   ├── control/           # Baseline
│   ├── treatment-1/       # First change
│   └── treatment-2/       # Optional
├── results/
│   ├── run-001.json       # Raw per-run data
│   └── ...
├── aggregate.json         # Computed stats
└── decision.md            # Keep / discard / modify / scale

experiment.md is the single source of truth. A fresh agent can read it and continue the experiment without context.

Sub-Agent Execution Patterns

Pattern A: Parallel Variant Testing Spawn N sub-agents, each implementing one variant. Collect results. Fastest when variants are independent.

Pattern B: Sequential Iteration Run → analyze → modify → run again. Use when each iteration depends on previous results (e.g., hill climbing).

Pattern C: Parameter Sweep Test a grid of parameters. Spawn parallel agents per cell. Good for config tuning (temperature, batch size, thresholds).

Pattern D: Cross-Domain Transfer Test the same intervention across multiple domains to measure generalizability.

Benchmark Script Discipline

The benchmark is the heart of the experiment. Requirements:

1. Must output at least one METRIC <name>=<numeric_value> line
2. Must exit 0 on success, non-zero on failure
3. Should be deterministic (same inputs → same measurement)
4. Should measure the thing, not the setup time
5. Should warm up caches/JIT before measuring if applicable
6. Should pin the tool version being measured (e.g. npx tool@0.6.81) — otherwise a new release of the tool can silently change output and invalidate comparisons

Measure the measurement tool before measuring the thing. If your benchmark has bugs (wrong regex, wrong units, broken timing, wrong exit-code interpretation), the experiment produces garbage and you won't know. Smoke-test the benchmark on a known input first. Run N=3 to confirm stddev is near zero before trusting any treatment effect.

When the measurement is deterministic, N=1 is enough. If 3 runs of the same variant produce identical scores (stddev=0), the tool is deterministic and the noise floor is below the effect size. Don't waste compute on N=10. Conversely, if stddev > 20% of the mean, you have noise — increase N, fix the benchmark, or accept the metric is unreliable and pick a different one.

Benchmark Script Pitfalls (hard-won)

1. date +%s%N is broken on macOS. BSD date has no nanosecond support and returns a literal N. Use gdate +%s%3N (Homebrew coreutils) or python3 -c "import time; print(int(time.time()*1000))". See scripts/benchmark-template.sh for the portable pattern.

2. grep -c returns multiline output when stderr mixes with stdout. Always pipe through head -1 | tr -d '\n' before passing to integer math. INSPECT_OVERFLOW_COUNT=0\n0 is a real failure mode.

3. tr -d ':1' is dangerous for parsing decimals. It eats leading 1s — 1.78 becomes .78. Use sed 's/:1$//' for known-suffix stripping, or better yet, use Python regex in a heredoc with env vars passed via export, not shell interpolation. Shell interpolation into Python source is fragile.

4. Pinning tool versions matters more than you think. npx tool@latest will silently change behavior when a new version ships. If you're comparing variants across days/weeks, pin the version (@0.6.81, @2.3.0, etc.) or your "regression" might be a tool upgrade, not a real effect. Conversely, if the tool is broken at your pinned version (e.g. hyperframes@0.6.81 has a missing-file bug), check if a newer patch version fixes it.

5. Output format assumptions are the #1 source of "the benchmark randomly gave 0.0 today." If you write MIN_CONTRAST=$(... | grep -oE "pattern" | head -1) and the tool changes its output format, the regex silently fails and MIN_CONTRAST becomes empty. Always log the raw excerpt to results (the template does this via raw_*_excerpt fields) so you can debug from the result file, not the missing output.

Example:

#!/bin/bash
cd "$1" || exit 1
# warmup
node test.js >/dev/null 2>&1
# measure
# IMPORTANT: `date +%s%N` is broken on macOS (BSD date returns literal "N").
# Use `gdate +%s%3N` (Homebrew coreutils) or
# `python3 -c "import time; print(int(time.time()*1000))"` for millisecond timing.
# `tr -d ':1'` is dangerous for parsing decimals — it eats leading 1s (1.78 → .78).
# Use `sed 's/:1$//'` to strip a known suffix, never `tr -d` on a multi-character
# pattern. `grep -c` returns multiline output if stderr mixes with stdout —
# always pipe through `head -1 | tr -d '\n'` before passing to integer math.
START=$(gdate +%s%3N 2>/dev/null || python3 -c "import time; print(int(time.time()*1000))")
node test.js
END=$(gdate +%s%3N 2>/dev/null || python3 -c "import time; print(int(time.time()*1000))")
DURATION=$((END - START))
echo "METRIC execution_time_ms=$DURATION"

Aggregation and Significance

After running all variants, aggregate:

python scripts/aggregate.py results/
# For metrics where lower is better (latency, error_rate, cost):
python scripts/aggregate.py results/ --lower-is-better latency_ms --lower-is-better cost_usd

This outputs aggregate.json with:

• Mean and stddev per variant
• Effect size (Cohen's d) — when noise is present
• Raw mean/relative difference — when measurement is deterministic (Cohen's d is undefined for zero variance)
• 95% confidence interval
• Verdict: "significant_improvement" / "significant_regression" / "no_significant_difference" / "insufficient_data"

Verdict rule:

• Deterministic measurement (pooled_std == 0): use relative difference. |rel_diff| ≥ 5% → significant improvement/regression; else no significant difference.
• Noisy measurement: signed Cohen's d (sign flipped for --lower-is-better metrics) > 0.5 → significant improvement, < -0.5 → significant regression, |d| < 0.2 → no significant difference.

Anti-Patterns

• "Try this and see if it's better" — without a baseline, you can't know
• Single-run conclusions — one data point is noise (unless you've confirmed the tool is deterministic with stddev=0; see Benchmark Script Discipline)
• Changing the metric mid-experiment — invalidates all previous data
• Optimizing without correctness checks — faster but wrong is worse than slower but right
• Fishing expeditions — running without a hypothesis finds false positives
• Asking "should I continue?" — the loop is autonomous. Decide based on numbers, not user input
• Confusing verbosity with rigor — writing more words in DESIGN.md / config / prompts does not move the metric. If a 5-line treatment and a 50-line treatment produce the same score, the verbosity is decoration. Find the single attribute or value that actually changed, and document that.
• Verifying by assertion instead of measurement — writing "this color pair is 8.2:1" in a comment is not verification. Running the validate tool is. Comments lie; measurement doesn't.

Relationship to Pi Autoresearch Extension

	This Skill (auto-experiment)	Pi Extension (autoresearch)
What	Strategy — how to think about experiments	Automation — tools that run the loop
Where	Main chat/runtime	Coding agent runtime
Tools	sessions_spawn, bash, write, edit	init_experiment, run_experiment, log_experiment
Use when	Multi-domain, human judgment needed, non-code metrics	Pure code optimization, user says "run autoresearch" in coding context

The three are complementary. This skill can design an experiment, then delegate the automated loop to the Pi extension if appropriate. Or it can run the entire experiment manually using sub-agents. The autoresearch skill provides the general framework if you need to reference the full 3-layer architecture.

Note: The autoresearch skill (nano/skills/autoresearch/) predates this skill and documents the general experiment framework including manual fallback when Pi tools are unavailable. It covers similar ground from a different angle. Use autoresearch for the general framework (especially when Pi tools might be involved); use auto-experiment for main-runtime orchestration with sub-agents.

Examples by Domain

Code Performance: Metric: execution_time_ms. Benchmark: shell timing. Variants: control (current code) vs treatment (optimized code).

Prompt Engineering: Metric: quality_score (rubric or human-rated). Benchmark: LLM evaluation script. Variants: different system prompts.

Config Tuning: Metric: throughput or accuracy. Benchmark: run system with parameters, measure. Variants: different parameter sets.

Business Metrics: Metric: conversion_rate or click_through_rate. Benchmark: A/B traffic splitting. Variants: different CTA copy or UI.

Agent Behavior: Metric: tool_first_rate (did correct tool fire on first response?). Benchmark: live test protocol with known triggers. Variants: different prompt structures.

Bundled Resources

• scripts/benchmark-template.sh — shell template for new benchmark scripts
• scripts/aggregate.py — statistical aggregation across runs (supports --lower-is-better for latency/cost metrics)
• scripts/parallel-runner.sh — helper for spawning parallel benchmark runs
• assets/experiment-bible-template.md — template for experiment.md
• tests/test_aggregate.py — regression tests covering Bug #1 (deterministic benchmarks) and direction handling

Changelog

• 1.0.1 — Fix Bug #1: aggregate.py now correctly handles deterministic benchmarks (zero variance) instead of returning insufficient_data for real effects. Added --lower-is-better flag for latency/cost/error metrics so verdict direction is correct. Added regression test suite.
• 1.0.0 — Initial release.

One-Liner

Experiment orchestration is the difference between "I think this is better" and "I ran 12 experiments across 3 variants and the effect size says it's 15% faster with 95% confidence." Use it when the metric matters and the space is too large to explore manually.