run-experiment

name

run-experiment

description

Main Agent orchestration skill for Phase 2 experiment execution — the step that takes fidelity-passed code and produces a result_{n}.md for the Analyst to interpret. Load when Phase 2 implementation has converged (every module's fidelity report is ALL CLEAR or accepted-with-override) and an experiment has not yet been launched; also load on Loop 3a re-entry after a code fix, or in Finalize mode when a prior Launch left result_{n}.md in RUNNING state and the run is now complete. Backend-agnostic: the skill defines the lifecycle contract (Launch / Finalize, sentinel-based completion detection, result file schema) but does not prescribe how commands are executed — local, SSH, SLURM, container, or any other backend the user has configured. Launches in background (via whatever detach mechanism the user's backend supports) so long runs survive /clear. Does NOT classify experiment failures — that is the Analyst's job downstream.

Phase 2 — Experiment Execution Pipeline

You are the Main Agent running an experiment. Your role is to execute the confirmed method in the user's configured execution environment, capture a faithful record of what happened, and write phase_2_verification/results/result_{n}.md — the artifact the Analyst will later interpret. You do not decide whether results are good, whether a failure is a code bug vs. an idea flaw, or whether to route to Loop 3a / Loop 3b. Those decisions belong to the Analyst and the Main Agent downstream of this skill.

What this skill is. A thin orchestrator around a deferred-execution launch, with two modes (Launch, Finalize), a smoke-test pre-flight, and a structured result file. It is not a training framework, not a metric analyzer, and not a routing engine.

What this skill is NOT.

• Not an environment installer — assumes the user has a working execution setup (local Python, remote SSH host, SLURM, Docker, etc.) that can run the Runner's invocation command.
• Not a backend prescriber — this skill defines the contract (where the log goes, what the sentinel file means, how background-survival is guaranteed) but not the implementation (ssh, screen, nohup, sbatch, etc.).
• Not a classifier — if the run fails, the skill records the failure and hands off; it does not say "code bug" vs. "implementation bug" vs. "idea flaw".

---

Execution environment contract

This skill assumes the user has configured an execution environment that satisfies four requirements. The specifics (local / remote SSH / SLURM / container / ...) are the user's choice; this skill does not prescribe them.

1. Runnability. The Runner's invocation command (from the Runner's README ## Invocation Command block) can be executed in the environment without further adaptation.
2. Log capture. Stdout + stderr of the run are captured to a file at phase_2_verification/results/log_{n}.txt, visible to the Main Agent's local filesystem (directly, or via a mount such as SSHFS, or via periodic copy — any means that makes the file readable locally).
3. Sentinel on exit. When the process exits, a one-line file phase_2_verification/results/log_{n}.done is written, containing exactly EXIT_CODE=<n> where <n> is the process's exit code. Finalize mode reads this sentinel.
4. Detached execution. The launch mechanism runs the process detached, so the run survives the Main Agent's session ending (/clear, terminal close, etc.). Examples of detach mechanisms the user might wire up: nohup, screen -dmS, tmux new -d, SLURM sbatch, Docker -d, Kubernetes Job.

The user documents how their environment satisfies these four points in whatever project-level configuration they maintain (e.g., CLAUDE.md, a separate runbook, or inline in the invocation brief). This skill halts and reports if the user has not set up an environment that satisfies the four-point contract above.

---

Preconditions

Before invoking this skill, every one of the following must hold. If any are missing, halt and report to the user; do not attempt to launch.

1. Spec exists and is confirmed. phase_1_design/method_design_v{n}.md exists with a matching ALL CLEAR phase_1_design/confirmer_report_{n}.md (or an accepted-with-override note from Phase 1 deadlock option (b)).
2. All modules implemented and fidelity-passed. Every module named in the spec's §Modules section has a phase_2_verification/code/{module}/ directory AND a corresponding phase_2_verification/fidelity/fidelity_{module}_v{r}.md with Verdict ALL CLEAR (or accepted-with-override per Phase 2 deadlock option (b)).
3. Runner module present. The last module in §Modules is the Runner (per method-architect convention). Its README at phase_2_verification/code/{runner}/README.md contains a fenced ## Invocation Command block with an exact shell command — no <...> placeholders. If the command has placeholders or the block is missing, halt and return to Phase 2's Coder round for the Runner (Case B entry).
4. Experiment setup locked in. phase_0_ideation/experiment_setup/EXPERIMENT_SETUP.md exists. The Runner's Dependencies must reference only resources audited there (or noted under Deviations). The spec-confirmer's Check C should have caught violations in Phase 1; re-verify here only as a sanity assertion, do not re-audit.
5. Execution environment configured. The user's environment satisfies the four-point contract above (runnability, log capture, sentinel on exit, detached execution). If the user has not told you how their environment satisfies these, halt and ask — do not guess a backend.
6. Working directories. Create if missing:
   • phase_2_verification/results/
   • phase_2_verification/analysis/ (created here so the Analyst can write there without a pre-step; this skill does not write into it).

---

Entry — Identify the mode

Before any action, determine the mode:

Mode: Launch (Case A / Case B) No phase_2_verification/results/result_*.md is in status: RUNNING, OR the invocation brief explicitly says "start new run" (Case B: Loop 3a re-launch after a code fix — the previous result file stays on disk as history, the new run gets the next {n}).

Mode: Finalize (Case C) At least one result_*.md is in status: RUNNING AND either (i) the PROJECT_STATE §2 check command for that run reports the process has exited (sentinel file exists), or (ii) the invocation brief explicitly says "finalize run {n}" (user override).

If both conditions hold (a prior run is still RUNNING AND a new Launch is requested), surface to the user and wait — do not auto-discard the running job and do not auto-merge its results.

---

Mode: Launch — Steps 1–8

Execute in order.

Step 1 — Verify preconditions

Walk the Preconditions list above. On any failure, report to the user with the specific failing item and halt. Do not attempt a partial launch.

Compute {n} — the next result index:

n = count(phase_2_verification/results/result_*.md) + 1

No padding, no reservations. {n} is the result identifier for the rest of this mode.

Step 2 — Parse the Runner

Read two files and extract the run parameters:

1. From phase_1_design/method_design_v{n}.md §Modules — the Runner subsection (the last module). Extract:

   • Slice — the one dataset, one LLM, method, 1–2 baselines from Dependencies.
   • Metric keys — the verbatim stdout/log keys from Outputs (e.g., eval/accuracy, eval/f1). The Analyst will grep for these.
   • Runtime class — if the Operation paragraph states one ("minutes" / "hours" / "days"); otherwise mark as "unknown".

2. From phase_2_verification/code/{runner}/README.md — the fenced ## Invocation Command block. Extract the command verbatim. Do NOT rewrite, do NOT substitute variables, do NOT strip newlines. This is the command that will run.

Announce the plan to the user before proceeding (Step 3 onward is visible action, so confirmation keeps the user in the loop):

Launching experiment {n}.

Slice: {dataset}, {llm}, method + {baselines}
Metric keys to capture: [list]
Expected runtime: {class or "unknown"}
Invocation command (verbatim from {runner}/README.md):
  {command block}

I will now smoke-test every module, then launch in the configured execution
environment. Proceed? (yes to continue, or name the step to skip — e.g.,
"skip smoke tests")

Wait for confirmation unless the invocation brief explicitly said "proceed without prompt". A user "skip smoke tests" response is valid and moves directly to Step 4.

Step 3 — Smoke-test pre-flight (default on)

For each module in spec-declared order, run its smoke_test.py in the user's configured execution environment. The exact command wrapper is environment-specific (python path/to/smoke_test.py locally; ssh alias "cd ... && python ..." for a remote; sbatch for SLURM; docker exec for a container; etc.). This skill treats the wrapper as opaque — the user's configured environment decides.

Rules:

• Run sequentially, not parallel. A smoke failure should be catchable at the earliest module.
• Capture exit code and last 20 log lines per smoke test.
• On any non-zero exit: halt. Report which module failed, the log tail, and stop. Do not proceed to the real run. Do NOT classify the failure — surface it to the user, and the user (or a follow-up phase2-implementation Case B invocation) addresses it.
• On all-pass: record "smoke: all passed" and proceed.

Skippable: if the brief or user said "skip smoke tests", write "smoke: skipped (per user)" and move to Step 4.

Step 4 — Resource pre-flight (optional, environment-specific)

If the execution environment exposes a resource whose availability matters before launch — GPU memory, SLURM queue slot, cluster quota, container-host capacity — check it now. This skill does not prescribe the check command; use whatever the user's environment supports (e.g., nvidia-smi locally or over SSH, squeue for SLURM, kubectl get nodes for Kubernetes). Record what was checked and the outcome.

Rules:

• If a resource is unavailable, halt and report. Ask the user whether to (a) wait, (b) proceed on a degraded resource, or (c) abort.
• If the check itself errors (tooling issue, host unreachable), halt — this is an infrastructure problem, not an experiment problem.
• Skippable: if the brief or user said "skip resource check", write "resource check: skipped (per user)" and move to Step 5.

Step 5 — Launch in background

Launch the Runner's invocation command in the user's configured execution environment. The launch must satisfy all four points of the execution environment contract — runnability, log capture to log_{n}.txt, sentinel log_{n}.done on exit, detached execution. Concretely, the launched process must:

• Execute the verbatim <runner_invocation_command> from Step 2, with nothing added and nothing removed from the command itself.
• Redirect combined stdout+stderr to phase_2_verification/results/log_{n}.txt (e.g., ... 2>&1 | tee phase_2_verification/results/log_{n}.txt).
• On exit, write phase_2_verification/results/log_{n}.done containing the literal line EXIT_CODE=<k> where <k> is the process's exit code.
• Run detached so the Main Agent's session can end without killing the run.

The wrapping mechanism is environment-specific — nohup, screen -dmS, tmux new -d, sbatch, docker run -d, Kubernetes Job, etc. Use the run_in_background: true flag on the Bash tool call that issues the launch, so the Main Agent's conversation is not blocked on the launch handshake.

Verify launch succeeded — the exact verification is environment-specific (e.g., screen -ls for screen, squeue -u $USER for SLURM, docker ps for docker). If verification fails within ~10 seconds, halt and report.

Step 6 — Write the initial result_{n}.md

Create phase_2_verification/results/result_{n}.md with status RUNNING:

# Experiment Result — Run {n}

**Status:** RUNNING
**Launched:** {ISO8601 timestamp}
**Spec:** phase_1_design/method_design_v{spec_n}.md
**Runner README:** phase_2_verification/code/{runner}/README.md
**Execution environment:** {short user-provided description — e.g., "local python3.11 in ./venv", "remote host 'gpu-3090' via ssh", "SLURM partition gpu on cluster X"}
**Detach mechanism:** {screen / nohup / tmux / sbatch / docker / ...}
**Log file:** phase_2_verification/results/log_{n}.txt
**Sentinel file:** phase_2_verification/results/log_{n}.done
**Runtime class (declared):** {minutes/hours/days/unknown}

## Slice

- Dataset: {dataset}
- LLM: {llm}
- Method: ours
- Baselines: {b1}, {b2}

## Declared metric keys

[Verbatim from the Runner's Outputs field — the Analyst will grep for these.]

- {key1}
- {key2}

## Invocation command

```bash
{verbatim runner invocation command}

Smoke-test pre-flight

{"All {N} modules passed" OR "Skipped per user" OR "Module {M} failed — launch aborted, see above"}

Resource pre-flight

{record of Step 4 outcome, or "Skipped per user"}

Completion

[Finalize mode populates this section. While status=RUNNING, it reads:]

Not yet finalized. To finalize, re-enter run-experiment in Finalize mode after the sentinel file log_{n}.done exists.

### Step 7 — Register in PROJECT_STATE §2

Append to `PROJECT_STATE.md` Section 2 (Running Experiments). Add exactly
one row:

```markdown
| Run | Started | Check command | On finish |
|-----|---------|---------------|-----------|
| {n} | {ISO8601} | `<user-provided shell command that exits 0 when phase_2_verification/results/log_{n}.done exists in the local filesystem>` | Re-enter `run-experiment` in Finalize mode with "finalize run {n}". |

The check command is the user's responsibility to supply at this point, because only the user knows which mount / sync / backend to inspect. A minimal example for a locally-visible sentinel (including via SSHFS mount) is test -f phase_2_verification/results/log_{n}.done; for a remote run without a local mount, something like ssh <alias> "test -f <remote_path>/phase_2_verification/results/log_{n}.done".

If Section 2 already has a header row, append under it; if it reads <!-- EMPTY ... -->, replace the EMPTY comment with the table.

session-start Step 2 reads this row and runs the check command on every session restore — that is the polling mechanism.

Step 8 — Report and exit

Single brief message to the user:

Experiment {n} launched.

Execution environment: {short description}
Detach mechanism: {mechanism}
Log: phase_2_verification/results/log_{n}.txt (updates live as the run writes)
Status file: phase_2_verification/results/result_{n}.md (status=RUNNING).
Runtime class: {class or "unknown"}.

The run will survive /clear. session-start will detect completion on next
restore via the check command in PROJECT_STATE §2. When done, re-enter
run-experiment with "finalize run {n}" — or let session-start queue it
for you.

I am not waiting on this run. Let me know if you want me to do something
else in the meantime.

Then stop. Do not poll, do not sleep, do not auto-re-check. The completion path is session-start's job.

---

Mode: Finalize — Steps 1–6

Execute in order.

Step 1 — Identify the run and verify completion

From the invocation brief, extract {n} (the run to finalize). Verify:

• phase_2_verification/results/result_{n}.md exists and its status is RUNNING.
• phase_2_verification/results/log_{n}.done exists (local readability is the contract — whether via direct local disk, SSHFS, or periodic sync is the execution environment's concern).

If the sentinel file is absent, the run has not completed. Halt and report — do not force-finalize a running job.

Read the sentinel:

cat phase_2_verification/results/log_{n}.done

Extract EXIT_CODE=<k>. This is the process's exit code.

Step 2 — Read the log

Read phase_2_verification/results/log_{n}.txt. If the file is very large (> ~5000 lines), capture the last 200 lines as the "tail"; otherwise the whole file is fine.

Step 3 — Extract declared metric keys

For each metric key declared in result_{n}.md's "Declared metric keys" section (populated in Launch Step 6), grep the log for the most recent occurrence:

grep -E "^{key}[=:\s]" phase_2_verification/results/log_{n}.txt | tail -1

Record the extracted value for each key. If a key does not appear in the log at all, record <not found in log>. Do NOT interpret — the Analyst does that.

Step 4 — Classify the run (mechanical only)

Based on the exit code:

• EXIT_CODE=0 → status: COMPLETE. Proceed to Step 5.
• EXIT_CODE≠0 → status: FAILED. Proceed to Step 5 — record the failure, do NOT diagnose.

This is the only classification this skill performs. "Code bug vs. implementation bug vs. idea flaw" is the Analyst's job.

Step 5 — Update result_{n}.md

Replace the ## Completion section with:

## Completion

**Status:** COMPLETE (or FAILED)
**Finalized:** {ISO8601 timestamp}
**Exit code:** {k}
**Wall-clock duration:** {computed from Launched → now}

### Extracted metric values

[One row per declared metric key, from Step 3.]

| Metric key | Last value in log |
|------------|-------------------|
| {key1} | {value1 or "<not found in log>"} |
| {key2} | {value2 or "<not found in log>"} |

### Log tail (last 200 lines)

{log tail}

[If the full log is ≤ 500 lines, include the full log instead and label
"### Full log" rather than "### Log tail".]

### Routing note (mechanical only)

{"No process-level failure — the Analyst should evaluate whether the
metric values support the Core Claims."
OR
"Process exited non-zero — the Analyst receives both the traceback (log
tail) and the partial metric values, and will classify per
DESIGN_NOTES §9."}

Also update the top-of-file **Status:** from RUNNING to COMPLETE or FAILED.

Step 6 — Deregister and hand off

Remove the run {n} row from PROJECT_STATE §2. If the table ends up empty, replace with <!-- EMPTY — no experiments currently running -->.

Report to the user:

Experiment {n} finalized.

Status: {COMPLETE or FAILED}
Exit code: {k}
Duration: {hh:mm:ss}
Extracted metric values: [list of key: value pairs, one line each]

Next step: invoke the `experiment-analyst` agent to interpret result_{n}.md.
I will not do that automatically — it is the Main Agent's routing decision,
and the Analyst's output shape matters for the downstream phase2-exit-check.

Then stop. Do NOT invoke the Analyst from inside this skill — the Main Agent triggers the Analyst as a separate, deliberate step.

---

Loop control

• No per-run cap. Each run-experiment invocation launches or finalizes exactly one run. Repeat invocations are how Loop 3a produces multiple result_{n}.md files over time.
• No timeout enforcement here. A run that has been RUNNING for days is a user-level concern — the check command in PROJECT_STATE §2 gives the user visibility. This skill does not kill long-running jobs.
• No concurrent runs of the same index. If Launch is invoked while result_{n}.md for the proposed {n} is already RUNNING, surface to the user — this should not happen under normal flow (Step 1 computes {n} from count + 1), but defensively check.

---

Files produced / touched by this skill

File	Written by	Notes
phase_2_verification/results/result_{n}.md	this skill	Launch writes with status=RUNNING; Finalize updates to COMPLETE/FAILED. The Analyst reads this; never overwrites.
phase_2_verification/results/log_{n}.txt	launched process (via whatever redirect the environment uses)	Must be locally readable (direct disk, SSHFS, sync). Raw stdout+stderr. This skill reads it in Finalize.
phase_2_verification/results/log_{n}.done	launched process (on exit)	One-line sentinel EXIT_CODE=<k>. Used by session-start's check command and Finalize's classification.
PROJECT_STATE.md §2	this skill	Launch adds a row; Finalize removes it.

Not written:

• Nothing under phase_2_verification/analysis/ — that is the Analyst.
• No changes to phase_1_design/, phase_0_ideation/, or phase_2_verification/code/.
• No edits to the confirmed spec, the fidelity reports, IDEA.md, or EXPERIMENT_SETUP.md.

---

What this skill does NOT do

• Does not classify failures as code / implementation / idea issues. The Analyst does that, per DESIGN_NOTES §9.
• Does not invoke the Analyst. The Main Agent triggers the Analyst after reading the Finalize report.
• Does not kill, restart, or retry failed runs. On a non-zero exit, the result file is finalized as FAILED and the user decides.
• Does not compare against prior results or baselines. The Analyst does that.
• Does not prescribe a backend. The execution environment contract defines what the skill expects; the user chooses any backend that satisfies it.
• Does not manage environments, dependencies, or secrets. All of that is out of scope; the skill assumes the backend is configured and halts if the four-point contract is not satisfied.
• Does not run Phase 3/4 main experiments. Phase 3/4 will need its own skill — possibly a thin extension of this one, but the slice discipline and small-scale framing are specific to Phase 2.

---

Interaction with other loops

• From phase2-implementation: enter Launch mode as Case A when every module's fidelity is ALL CLEAR and the user confirms "run the experiment".
• From Loop 3a (code-fix retry): enter Launch mode as Case B after phase2-implementation has re-converged for the fixed module. A new {n} is allocated; the prior result file stays on disk.
• From session-start: on next session after a background Launch, if session-start's Step 2 check command shows the sentinel exists, it queues "finalize run {n}" as Next Steps item 1. The Main Agent then re-enters this skill in Finalize mode.
• To experiment-analyst: after Finalize, the Main Agent invokes the Analyst on result_{n}.md. This skill does NOT invoke it.
• To phase2-exit-check: further downstream of the Analyst. Not this skill's concern.