experiment-setup-pipeline

name

experiment-setup-pipeline

description

Main Agent orchestration skill for the Experimental Setup Module — the pre-Phase-1 step that audits which datasets, metrics, baselines, LLMs, and reusable code are actually available for the idea. Load when Loop 2 has passed and EXPERIMENT_SETUP.md has not yet been produced, or when the previous audit flagged thin coverage and the Main Agent needs to drive a supplementary scout search and re-audit. Drives the auditor → exit-check → optional supplement cycle and hands off to Phase 1 (Method Design).

Experimental Setup Module — Pipeline

You are the Main Agent running the Experimental Setup Module. Your role is to invoke the auditor, evaluate its output via the exit check, and either hand off to Phase 1 or drive a supplementary scout search and re-audit. You do not perform the audit yourself.

Do not attempt to fix gaps in the audit. If a category came up empty, the remedy is either a supplementary literature search (orchestrated by you, executed by the Scout) or — at the user's call — relaxing the IDEA's scope. The auditor's job is to report; yours is to route.

---

Entry — Identify the case

Before Round 1, determine which entry case applies:

Case A — Fresh entry from Loop 2 Loop 2 has just passed and no EXPERIMENT_SETUP.md exists yet. There are no prior audits or supplementary scout reports tied to this module.

Case B — Re-audit after supplement A previous audit returned NEEDS-REWORK because one or more categories were thin. The Main Agent has already invoked the Scout (in "Experimental Setup Audit" context) and produced a supplementary scout_report_{n}.md in phase_0_ideation/literature/. You now re-invoke the auditor.

---

Round structure

Execute Steps 1–4 in order. Do not skip steps.

Step 1 — Confirm preconditions

Verify the following before invoking the auditor:

• phase_0_ideation/ideas/IDEA.md exists.
• At least one phase_0_ideation/stress_test/tester_report_*.md exists (signals Loop 2 has run).
• The most recent Loop 2 exit check passed, OR an override_note.md exists in phase_0_ideation/stress_test/ (user explicitly chose to proceed despite Loop 2 flags).

If any precondition is missing, halt and report to the user. Do not silently proceed.

Also create the output directory if it does not exist: phase_0_ideation/experiment_setup/.

Step 2 — Invoke the auditor

Invoke the experiment-setup-auditor agent with a brief.

Case A brief:

Audit experimental setup for the idea at phase_0_ideation/ideas/IDEA.md.
Source literature: every scout_report_*.md in phase_0_ideation/literature/
and every tester_literature_*.md in phase_0_ideation/stress_test/.
Output: phase_0_ideation/experiment_setup/EXPERIMENT_SETUP.md

Case B brief:

Re-audit experimental setup for the idea at phase_0_ideation/ideas/IDEA.md.
A supplementary scout report was added at phase_0_ideation/literature/scout_report_{n}.md
to address thin coverage of: [category, e.g., "datasets"].
Re-include all source literature in your aggregation.
Output: phase_0_ideation/experiment_setup/EXPERIMENT_SETUP.md (overwrite)

The auditor overwrites EXPERIMENT_SETUP.md each time — it is a single living document, not versioned. Historical audits, if needed, are captured in _logs/ (not your concern at this step).

Wait for the auditor to finish. Confirm EXPERIMENT_SETUP.md exists at the expected path before continuing.

Step 3 — Evaluate

Run the experiment-setup-exit-check skill against the produced EXPERIMENT_SETUP.md.

If PASS: the audit is complete. Surface the result to the user:

Experimental Setup Module — PASSED

EXPERIMENT_SETUP.md is ready for Phase 1 (Method Design).

Selected setup:
- Datasets: [list names from the file]
- Baselines: [list names from the file]
- LLM(s): [list]
- Reusable code: [count of repos, or "none identified"]

Items removed during verification: [count from Removed/Unavailable section, or "none"]
[If count > 0, summarize the most consequential removals in 1–2 lines so the
user knows what was dropped.]

The next step is Phase 1: Method Design.

Then proceed to handoff (see "Handoff to Phase 1" below).

If NEEDS-REWORK: go to Step 4.

Step 4 — Supplement decision and user notification

When the exit check returns NEEDS-REWORK, the most common reason is that one or more categories ended up empty (or nearly empty) after the auditor's verification. You do not auto-fix this.

Notify the user with the exit-check's findings and your recommended action:

Experimental Setup Module — NEEDS-REWORK

The audit produced a setup, but the exit check flagged the following gaps:
[list from exit check, e.g., "Datasets: zero verified entries — source literature
yielded only X (gated, no access) and Y (404)"]

Options:
(a) Supplement — I will invoke the Literature Scout in Experimental Setup Audit
    context, scoped to the thin category. The Scout will search for additional
    candidates with public availability. Then I re-invoke the auditor.
(b) Accept and proceed — the gap is acceptable; the Architect will handle the
    missing category at Phase 1 (or you intend to address it manually). I will
    proceed to handoff with the current EXPERIMENT_SETUP.md.
(c) Revise the idea — the gap suggests the IDEA's chosen evaluation is infeasible.
    I will return to Loop 1 with this as the standing requirement (e.g., "revise
    Problem Scope to use evaluations with available data").
(d) Abandon — discard this idea.

Wait for user instruction. Do not auto-supplement.

On user response:

• (a) Supplement: invoke the literature-scout agent with the "Experimental Setup Audit" context. Brief:

  Survey datasets/baselines/metrics/code for [task framing from IDEA's Problem Scope].
  Scope: focus on [thin category, e.g., "publicly available datasets for X task"].
  Output: phase_0_ideation/literature/scout_report_{n}.md
  

  where {n} is the next available scout report version. Then re-enter this pipeline as Case B.
• (b) Accept and proceed: write a short note at the top of phase_0_ideation/experiment_setup/EXPERIMENT_SETUP.md recording the user's acceptance ("Accepted with known gaps in [category] on [date] — Architect to resolve at Phase 1"). Proceed to handoff.
• (c) Revise the idea: invoke loop1-ideation-pipeline with Case C entry and the gap as the standing requirement. When Loop 1 + Loop 2 complete, re-enter this pipeline as Case A.
• (d) Abandon: stop. Record the abandonment reason in phase_0_ideation/experiment_setup/abandon_note.md.

---

Loop control

Hard cap: 2 supplementary rounds.

A "supplementary round" is one Scout supplement + one re-audit. If after 2 supplementary rounds the exit check still returns NEEDS-REWORK, do not continue automatically. Surface the situation to the user:

Experimental Setup Module has completed 2 supplementary rounds without resolving
the flagged gaps.

Persistent gaps:
[list from latest exit check]

Options:
(a) Continue — confirm to attempt another supplement.
(b) Accept and proceed — same as option (b) in Step 4.
(c) Revise the idea — same as option (c) in Step 4.
(d) Abandon — discard this idea.

Wait for user instruction before acting.

---

Handoff to Phase 1

When the audit passes (or the user accepts current gaps), the module is complete. Phase 1 (Method Design) is not yet built — for now, simply announce:

Phase 0 is complete. The Architect agent and Phase 1 pipeline are not yet
implemented. To resume, the next session should pick up at Phase 1 design once
those are in place.

Phase 0 outputs ready for Phase 1:
- phase_0_ideation/ideas/IDEA.md
- phase_0_ideation/experiment_setup/EXPERIMENT_SETUP.md
- phase_0_ideation/stress_test/tester_report_{latest}.md (for known-risk reference)

The Phase 0 summary file (phase_0_ideation/PHASE_0_SUMMARY.md, per file_organization_structure.md) is the natural next artifact — when Phase 1 is built, that skill should write the summary as part of its entry. Do not write it here.

---

Files produced by this module

File	Written by	Notes
phase_0_ideation/experiment_setup/EXPERIMENT_SETUP.md	Auditor	Overwritten on each audit; the single living artifact for this module
phase_0_ideation/literature/scout_report_{n}.md	Scout (when supplemented)	Numbered globally per the same counter Loop 1 uses; never overwritten
phase_0_ideation/experiment_setup/abandon_note.md	Main Agent	Only written if the user chooses (d) Abandon

Why EXPERIMENT_SETUP.md is overwritten: the audit is a single feasibility snapshot tied to the current IDEA.md. If the idea changes (Loop 1 retreat from this module), the audit must be re-run from scratch — historical audits do not need to be preserved in the working tree. If a record is wanted, log it to _logs/.

---

Interaction with other loops

• From Loop 2: enter as Case A after Loop 2 PASS.
• To Loop 1: only on user choice (c) Revise the idea — the IDEA's Problem Scope is the most likely revision target, since the audit specifically tests whether the proposed evaluation is grounded in obtainable resources.
• To Phase 1: handoff per "Handoff to Phase 1" above. Phase 1 reads both IDEA.md and EXPERIMENT_SETUP.md as input.