phase3-main-experiment

name

phase3-main-experiment

description

Main Agent orchestration skill for Phase 3 — scaling the Phase-2-validated minimum-viable slice up to the full EXPERIMENT_SETUP.md matrix (all datasets × all LLMs × all baselines + method) that a paper's Main Results table needs. Load when phase2-exit-check PASSed with dominant rollback depth None, and Phase 3 has not yet produced a fully analyzed matrix; also load on re-entry after Loop 4 supplementary experiments. Drives: (i) spec upgrade from method_design_v{n}.md to framework_v{m}.md via method-architect ↔ spec-confirmer, (ii) code upgrade via the per-module Coder ↔ Fidelity loop, (iii) matrix execution via multiple run-experiment calls, (iv) per-cell experiment-analyst invocation, (v) story-coherence-advisor checkpoints. Hands off to phase3-exit-check. Does NOT run ablations — that is Phase 4.

Phase 3 — Main Experiment Pipeline

You are the Main Agent running Phase 3. Your role is to take the Phase-2-validated slice and expand it into the full experimental matrix declared in EXPERIMENT_SETUP.md, so that the project has the Main Results table a paper needs to defend IDEA.md's Core Claims. You orchestrate four sub-processes in order: spec upgrade, code upgrade, matrix execution, and narrative checkpoints. You do not write specs or code yourself, do not interpret results, and do not decide whether Phase 3 is done — those are agent / skill responsibilities downstream.

Scope boundary. Phase 3 produces Main Results: the numbers that go in the main-results table of the paper, plus the supporting analyses and the evolving narrative. Phase 4 (Ablation) runs in parallel or after Phase 3 and has its own skill (phase4-ablation, not yet built); you do not invoke it. If Phase 3 execution surfaces a question that ablation would answer, note it for the Phase 4 entry — do not address it here.

Transparent-relay rule. Between agents (Architect / Confirmer / Coder / Fidelity Checker / Analyst / Story Advisor) you are a pipe, not a participant. You do not rewrite their outputs, do not add your own assessments between rounds, do not pre-filter findings. Your active decisions are: case entry, precondition checks, loop convergence detection, matrix enumeration, and deadlock escalation.

---

Entry — Identify the case

Before starting, determine which entry case applies:

Case A — Fresh entry from Phase 2 exit phase_2_verification/analysis/analysis_{n}.md (latest) had dominant rollback depth None, phase2-exit-check returned PASS, and no Phase 3 artifacts yet exist (phase_3_main_experiment/specs/ is empty). This is the normal path.

Case B — Re-entry after Loop 4 supplementary experiments Phase 5/6 review raised major issues, the Main Agent chose to supplement experiments rather than rewrite the paper, and one or more new matrix cells or new baselines need to be added. A framework_v{m}.md already exists; the invocation brief names the specific additions (new cells, new baselines, new dataset variants). Do NOT re-upgrade the full spec — scope the upgrade to the additions.

Case C — Re-entry after a Phase 3 rollback (via phase3-exit-check) phase3-exit-check returned NEEDS-REWORK with a non-None rollback depth, the Main Agent chose to address a Phase 3-internal issue (code bug in a single Phase 3 module, failed cell that needs rerun), and a targeted sub-process is re-entered. The brief names what to redo.

If no case matches — e.g., Phase 2 never passed, or framework_v{m}.md exists but Phase 3 has not been declared started — halt and ask the user which case applies. Do not guess.

---

Preconditions (all cases)

Verify before any sub-process invocation:

1. Phase 2 has passed. Latest phase_2_verification/analysis/analysis_{n}.md has dominant rollback depth None, and a phase2-exit-check PASS block is in the session log or the analysis file's preamble.
2. Experiment setup is locked. phase_0_ideation/experiment_setup/EXPERIMENT_SETUP.md exists and contains a matrix definition — at minimum: a Datasets table, an LLMs table, and a Baselines table. If any table is missing, halt and route to experiment-setup-pipeline (supplementary search mode) — do not attempt Phase 3 against a partial matrix.
3. Phase 1 spec exists. phase_1_design/method_design_v{n}.md with its ALL CLEAR confirmer report (or accepted override) is the starting point for the Architect's scope expansion.
4. Execution environment configured. Phase 3's matrix execution invokes run-experiment many times; each call will re-verify the four-point execution contract, but you check once upfront to halt early if the environment is unconfigured.
5. Working directories. Create if missing:
   • phase_3_main_experiment/specs/
   • phase_3_main_experiment/code/
   • phase_3_main_experiment/fidelity/
   • phase_3_main_experiment/results/
   • phase_3_main_experiment/analysis/
   • phase_3_main_experiment/story_checks/

On any precondition failure, halt and report the specific failing item.

---

Sub-process 1 — Spec upgrade (Architect ↔ Confirmer)

Case A: run in full. Case B: run scoped to the named additions. Case C: skip unless the brief names a spec issue.

Step 1 — Invoke the Architect (Case B entry)

Compute {m} — the next framework spec index:

m = count(phase_3_main_experiment/specs/framework_v*.md) + 1

Invoke method-architect with Case B (scope-expansion) brief:

Case: Phase 3 scope expansion.
Source spec: phase_1_design/method_design_v{n}.md (Phase 2's confirmed spec)
Experiment setup: phase_0_ideation/experiment_setup/EXPERIMENT_SETUP.md
Phase 2 analyses to read: phase_2_verification/analysis/analysis_*.md
Output: phase_3_main_experiment/specs/framework_v{m}.md
Round: 1 (initial framework draft)

Your task: expand the minimum-viable slice (1 dataset × 1 LLM × method +
1–2 baselines) into the full matrix declared in EXPERIMENT_SETUP.md. The
Runner module's §Dependencies and §Operation must be updated to iterate
the full matrix; its §Outputs must declare the logged metric keys for
every matrix cell (not just the slice cell). Every non-Runner module may
or may not need revision — if a module's behavior is matrix-invariant,
carry it forward unchanged and note so explicitly.

[Case B only: The Main Agent is supplementing — scope the expansion to
these additions: {list from invocation brief}. Do not re-open other
modules.]

Wait for the Architect to finish. Confirm framework_v{m}.md exists.

Step 2 — Invoke the Confirmer

Invoke spec-confirmer against the new framework:

Spec to audit: phase_3_main_experiment/specs/framework_v{m}.md
Experiment setup: phase_0_ideation/experiment_setup/EXPERIMENT_SETUP.md
IDEA: phase_0_ideation/ideas/IDEA.md
Round: 1
Output: phase_3_main_experiment/specs/confirmer_report_{m}_v1.md

Wait for the Confirmer. Read the Verdict from the report.

Step 3 — Architect ↔ Confirmer loop (max 3 rounds)

If Verdict = ALL CLEAR: framework is confirmed. Proceed to Sub-process 2.

If Verdict = REVISIONS NEEDED: invoke the Architect for revision round r ← r + 1 with the Confirmer's findings as the brief. Then re-invoke the Confirmer. Repeat until ALL CLEAR or r > 3.

If r > 3 with still-open findings: escalate with the same deadlock menu as phase1-method-design (continue / accept-with-override / abandon Phase 3). Do not auto-route.

When the loop converges on ALL CLEAR (or user-accepted override), record the confirmed framework version and move to Sub-process 2.

---

Sub-process 2 — Code upgrade (Coder ↔ Fidelity per module)

The Phase 2 code under phase_2_verification/code/ implemented the slice. Phase 3's matrix expansion usually affects the Runner (now iterates the matrix) and may affect one or two other modules (e.g., a data loader that now supports multiple datasets). Modules whose behavior is matrix-invariant can be carried forward unchanged.

Step 1 — Enumerate modules and classify

Read framework_v{m}.md §Modules. For each module, compare against the Phase 2 method_design_v{n}.md version:

• Unchanged: spec is byte-identical except for cross-references. Copy the Phase 2 code directory to phase_3_main_experiment/code/{module}/ verbatim (git cp or equivalent); mark as carry-forward. No Coder invocation.
• Changed: spec diff exists. Re-implement via Coder ↔ Fidelity.
• New: named in framework but not in Phase 2. Implement fresh via Coder ↔ Fidelity.

Announce the plan to the user before starting:

Phase 3 code upgrade plan:

Carry-forward (no Coder work):
  - {module_a}: unchanged
  - {module_b}: unchanged
Re-implement (Coder ↔ Fidelity):
  - {module_c}: {one-line diff summary}
  - {runner}: matrix iteration
New:
  - {module_d}: {role}

Per-module cap: 3 Coder ↔ Fidelity rounds. Carry-forward modules are not
audited. I will process Re-implement and New modules in spec-declared
order.

Wait for user confirmation unless the brief said "proceed without prompt".

Step 2 — Run the Coder ↔ Fidelity loop per non-carry-forward module

For each Re-implement or New module, run the same loop shape as phase2-implementation Steps 3–6, but with paths rooted at phase_3_main_experiment/:

• Code directory: phase_3_main_experiment/code/{module}/
• Fidelity reports: phase_3_main_experiment/fidelity/fidelity_{module}_v{r}.md
• Spec reference: phase_3_main_experiment/specs/framework_v{m}.md

Every other aspect of the per-module loop — round cap, deadlock escalation menu, cross-module I/O handling — is identical to phase2-implementation's Steps 3–6 and its "On per-module deadlock" section. If you detect behavior that diverges between Phase 2 and Phase 3 requirements, surface to the user and ask — do not silently adapt.

Step 3 — Exit condition

When every module has either (i) carry-forward status or (ii) an ALL CLEAR Fidelity Report for Phase 3 (or user-accepted override), Sub-process 2 is done. Announce:

Phase 3 code upgrade — COMPLETE

Carry-forward modules: [list]
Re-implemented / new modules with Phase 3 fidelity ALL CLEAR:
  - {module_c}: Round {r_c}
  - {runner}: Round {r_r}
  - ...

Accepted additions (if any): [list]

The Runner's README at
phase_3_main_experiment/code/{runner}/README.md now carries the Phase 3
Invocation Command (iterates the full matrix). Proceeding to matrix
execution.

---

Sub-process 3 — Matrix execution (run-experiment, once per cell)

Step 1 — Enumerate the matrix

Read framework_v{m}.md §Modules §Runner §Dependencies. The Runner names the full matrix — which datasets, which LLMs, which methods/baselines. Read EXPERIMENT_SETUP.md to cross-check that every matrix entry is audited (the Confirmer should have enforced this upstream; re-verify as sanity only).

The matrix has N cells. The Runner's §Operation paragraph declares how cells are iterated — three possibilities:

1. Single-process matrix Runner — one run-experiment call iterates the full matrix in-process, and the log emits every metric key for every cell. Used when cells are cheap and can share a Python process (most common for multi-dataset RAG evaluations on the same LLM).
2. Per-cell Runner — the Runner exposes a CLI flag (--cell) and one run-experiment call is launched per cell. Used when cells differ in LLM (different model loads) or have very different runtimes.
3. Hybrid — the Runner iterates some axes in-process (e.g., datasets) but others (e.g., LLMs) require separate launches.

The Architect is instructed to pick one and state it explicitly in the Runner's §Operation. If §Operation is ambiguous about iteration strategy, halt and re-invoke the Architect ↔ Confirmer loop — do not decide for them.

Step 2 — Plan the launch sequence

Produce a launch table:

Launch	Cell(s) covered	Expected runtime class	Metric keys to capture
1	...	hours / days	[keys]
...	...	...	...

Announce to the user before the first launch:

Phase 3 matrix execution plan:

Iteration strategy: single-process / per-cell / hybrid
Total launches to make: {K}
Total expected runtime (sum of class estimates): {rough range}

Launch table:
{table above}

I will launch each via run-experiment, wait for the sentinel, invoke
experiment-analyst, then (at the checkpoints below) invoke
story-coherence-advisor. I will NOT auto-parallelize launches that share
the same GPU — per-cell launches are serial unless you tell me otherwise.

Story Advisor checkpoints:
  - After launch 1 analysis (Context A → first narrative draft anchored
    in Phase 3 evidence)
  - After every 3rd launch analysis thereafter (Context B)
  - After the final launch analysis (Context B, final pass)

Proceed?

Wait for user confirmation. A "proceed with different checkpoint cadence" response is valid — update the plan and re-announce.

Step 3 — Execute one launch

Invoke run-experiment in Launch mode. Pass the brief pointing at the Phase 3 paths:

Phase: 3 (main experiment)
Spec: phase_3_main_experiment/specs/framework_v{m}.md
Runner code: phase_3_main_experiment/code/{runner}/
Results directory: phase_3_main_experiment/results/
Launch scope: {cell(s) per the launch-table row}

run-experiment will handle the background + sentinel contract in the user's configured execution environment. It writes phase_3_main_experiment/results/result_{n}.md with status RUNNING, registers in PROJECT_STATE §2, and returns.

Do not wait synchronously. Return control to the user as run-experiment instructs. session-start will poll on every restore. When the sentinel drops, re-enter run-experiment in Finalize mode for that run (either automatically on session restore or via user instruction).

Step 4 — Per-run Analyst invocation (after Finalize)

Once run-experiment Finalize has written result_{n}.md with status COMPLETE or FAILED, invoke experiment-analyst:

Result to analyze: phase_3_main_experiment/results/result_{n}.md
Spec: phase_3_main_experiment/specs/framework_v{m}.md
IDEA: phase_0_ideation/ideas/IDEA.md
Output: phase_3_main_experiment/analysis/analysis_{n}.md

The Analyst's methodology is unchanged from Phase 2. Its Claim-Result Comparison table now covers the cells this run touched. If a run's dominant rollback depth is non-None, halt the matrix execution loop and escalate per §Matrix-level failure handling below — do not continue launching cells against a broken method or broken code.

Step 5 — Story Advisor checkpoint (when the plan says so)

After analyses that the Step 2 plan marked as a checkpoint, invoke story-coherence-advisor:

Invocation context: B (Phase 3 checkpoint)
Inputs:
  - IDEA: phase_0_ideation/ideas/IDEA.md
  - Spec: phase_3_main_experiment/specs/framework_v{m}.md
  - Phase 2 analyses: phase_2_verification/analysis/analysis_*.md
  - Phase 3 analyses so far: phase_3_main_experiment/analysis/analysis_*.md
  - Prior story_check: phase_3_main_experiment/story_checks/story_check_{k-1}.md
                       (or "none" for the first Phase 3 checkpoint — use
                        Context A instead)
Output: phase_3_main_experiment/story_checks/story_check_{k}.md

Read the Verdict. If COHERENT, continue matrix execution (loop back to Step 3 for the next launch). If FRACTURED, do NOT auto-continue; surface the Story Advisor's Most-important-reconciliation to the user and wait:

Phase 3 Story Check — FRACTURED at launch {n}.

Most important reconciliation: {one line from story_check_{k}.md}

Reconciliation options the Story Advisor proposed:
{list}

Options for you:
(a) Continue matrix execution — note the fracture and let downstream
    analyses + ablation sharpen attribution. Appropriate when the
    fracture is a known attribution gap, not an evidence failure.
(b) Pause and reframe IDEA — route to Refiner (Loop 1 re-entry,
    Case C). Appropriate for story-ahead-of-method fractures where the
    claim is the drift.
(c) Pause and extend method — route to Architect (Loop 3b scoped).
    Appropriate for method-ahead-of-story fractures where the spec
    itself needs to name the new behavior.
(d) Pause and request supplementary experiments — escalate to Phase 4
    setup or a new matrix cell. Appropriate for "dataset generality"
    PARTIAL fractures.
(e) Abandon Phase 3 and retreat to Phase 0 — reserved for fractures
    that imply the idea itself is unsound. Rare at this stage.

Wait for user instruction. Do not auto-continue on a FRACTURED verdict.

Step 6 — Matrix exit

When every cell has a finalized result_{n}.md + analysis_{n}.md, and the final Story Advisor invocation (Context B, final pass) has run, Sub-process 3 is complete. Move to Sub-process 4.

---

Sub-process 4 — Exit

Invoke phase3-exit-check against the accumulated Phase 3 artifacts. It returns PASS or NEEDS-REWORK with a routing-advisory table.

On PASS: announce handoff:

Phase 3 — COMPLETE

Spec: phase_3_main_experiment/specs/framework_v{m}.md
Cells covered: {N}
Analyses: phase_3_main_experiment/analysis/analysis_1.md … _{N}.md
Story check verdicts (chronological): {list of COHERENT / FRACTURED}
Latest story check: phase_3_main_experiment/story_checks/story_check_{k}.md
  (Verdict: {COHERENT / FRACTURED (scoped)})

Hand-off: Phase 4 (Ablation) is the next phase. The phase4-ablation skill
is NOT YET BUILT. When it is, it will take framework_v{m}.md + the final
story_check as inputs and design the ablation plan.

Then stop.

On NEEDS-REWORK: Main Agent reads the advisory table and picks a case per §Matrix-level failure handling below, or re-invokes this skill as Case C.

---

Matrix-level failure handling

A cell's analysis may return a non-None rollback depth mid-matrix (Sub-process 3 Step 4). Handle per depth:

Depth	Action
Low (code bug in one module)	Halt matrix. Route to phase2-implementation Case B scoped to Phase 3 code directory, fix the named module, then re-invoke this skill as Case C to rerun affected cells.
Medium (bad setup / feasibility)	Halt matrix. Route to experiment-setup-pipeline supplementary mode (to revise the relevant resource) OR phase1-method-design Case B if a spec tweak is needed. Resume as Case C after the upstream fix.
High (method component fails / claim mismatch in main results)	Halt matrix. Surface to user. The deeper rollback (Loop 3b in Phase 2's vocabulary) may be the right call, but in Phase 3 it is significant — a failed main result implies the method does not generalize beyond the Phase 2 slice. Options: (i) reframe claim via Refiner (Loop 1 Case C), (ii) redesign component via Architect (Phase 1 Case B), (iii) accept narrower claim scope + continue matrix. Wait for user.
Critical (idea fundamentally invalid in main results)	Halt matrix. Surface to user immediately. This is Phase 0 re-ideation territory; do NOT auto-route, user confirmation required.

Do not silently continue after a non-None-depth cell. The matrix is not salvaged by averaging over a failed cell; the Main Results table needs defensible cells.

---

Loop control

• Architect ↔ Confirmer cap (Sub-process 1): 3 rounds.
• Coder ↔ Fidelity cap per module (Sub-process 2): 3 rounds.
• Matrix execution (Sub-process 3): no round cap — every cell must be attempted. A cell's non-None analysis is a loop-halt signal per §Matrix-level failure handling, not a retry.

---

Files produced by this pipeline

File	Written by	Notes
phase_3_main_experiment/specs/framework_v{m}.md	Architect	Full-matrix spec. Versioned.
phase_3_main_experiment/specs/confirmer_report_{m}_v{r}.md	Confirmer	One per Architect ↔ Confirmer round.
phase_3_main_experiment/code/{module}/	Coder (or carry-forward copy)	Main source per module.
phase_3_main_experiment/code/{module}/smoke_test.py	Coder	NOT executed by this skill — run-experiment handles smoke tests.
phase_3_main_experiment/code/{runner}/README.md	Coder	Contains the Phase 3 Invocation Command block.
phase_3_main_experiment/fidelity/fidelity_{module}_v{r}.md	Fidelity Checker	One per Coder ↔ Fidelity round per module.
phase_3_main_experiment/results/result_{n}.md	run-experiment	One per launch. Status transitions RUNNING → COMPLETE/FAILED.
phase_3_main_experiment/results/log_{n}.txt	run-experiment (via execution environment)	Live log, locally readable per the execution contract.
phase_3_main_experiment/results/log_{n}.done	launched process on exit	Sentinel. run-experiment Finalize reads.
phase_3_main_experiment/analysis/analysis_{n}.md	experiment-analyst	One per finalized run.
phase_3_main_experiment/story_checks/story_check_{k}.md	story-coherence-advisor	One per checkpoint. First (post-Phase-2) is Context A; subsequent are Context B.
phase_3_main_experiment/PHASE_3_SUMMARY.md	Main Agent	Written at handoff on PASS. One-page rollup: framework version, cells run, final story check verdict.

---

What this skill does NOT do

• Does not run ablations. Phase 4 (phase4-ablation, not yet built) handles ablation. If the Story Advisor proposes an ablation-style reconciliation, surface it as a deferred item for Phase 4 — do not design or run ablations here.
• Does not write the paper. Contribution bullets produced by the Story Advisor are narrative outlines, not sections. The Writer (Phase 6) expands them.
• Does not second-guess the Analyst or the Story Advisor. Their outputs are authoritative for the loop control decisions this skill makes. If you believe either is wrong, re-invoke them with a specific gap named; do not override.
• Does not decide Loop 3 routing. Non-None rollback depths trigger the §Matrix-level failure handling menu, which surfaces to the user; the Main Agent decides the route.
• Does not parallelize launches by default. GPU sharing is a user decision, not a skill default.

---

Interaction with other loops

• From Phase 2 (phase2-exit-check): enter as Case A when depth = None and user confirms Phase 3 start.
• To Phase 2 (phase2-implementation Case B): on matrix-level Low depth.
• To Phase 1 (phase1-method-design Case B): on matrix-level Medium (spec tweak) or High (component redesign).
• To Experimental Setup (experiment-setup-pipeline): on matrix-level Medium (resource revision).
• To Phase 0 (loop1-ideation-pipeline Case C): on matrix-level Critical, user-confirmed only.
• From Loop 4 (review supplementary): enter as Case B with additions named.
• To Phase 4 (phase4-ablation, not yet built): via the Sub-process 4 PASS handoff.