Workflow 1.5: Bridge between idea discovery and auto review. Reads EXPERIMENT_PLAN.md, implements experiment code, deploys to GPU, collects initial results. Use when user says "ๅฎ็ฐๅฎ้ช", "implement experiments", "bridge", "ไป่ฎกๅๅฐ่ทๅฎ้ช", "deploy the plan", or has an experiment plan ready to execute.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
A direct command skips the review prompt. Inspect the source before running it.
Workflow 1.5: Bridge between idea discovery and auto review. Reads EXPERIMENT_PLAN.md, implements experiment code, deploys to GPU, collects initial results. Use when user says "ๅฎ็ฐๅฎ้ช", "implement experiments", "bridge", "ไป่ฎกๅๅฐ่ทๅฎ้ช", "deploy the plan", or has an experiment plan ready to execute.
Implement and deploy experiments from plan: $ARGUMENTS
Overview
This skill bridges Workflow 1 (idea discovery + method refinement) and Workflow 2 (auto review loop). It takes the experiment plan and turns it into running experiments with initial results.
CODE_REVIEW = true โ GPT-5.6-Sol xhigh reviews experiment code before deployment. Catches logic bugs before wasting GPU hours. Set false to skip.
AUTO_DEPLOY = true โ Automatically deploy experiments after implementation + review. Set false to manually inspect code before deploying.
SANITY_FIRST = true โ Run the sanity-stage experiment first (smallest, fastest) before launching the rest. Catches setup bugs early.
MAX_PARALLEL_RUNS = 4 โ Maximum number of experiments to deploy in parallel (limited by available GPUs).
BASE_REPO = false โ GitHub repo URL to use as base codebase. When set, clone the repo first and implement experiments on top of it. When false (default), write code from scratch or reuse existing project files.
COMPACT = false โ When true, (1) read idea-stage/IDEA_CANDIDATES.md instead of full idea-stage/IDEA_REPORT.md if available, (2) append experiment results to EXPERIMENT_LOG.md after collection.
Override: /experiment-bridge "EXPERIMENT_PLAN.md" โ compact: true, base repo: https://github.com/org/project
Inputs
This skill expects one or more of:
refine-logs/EXPERIMENT_PLAN.md (best) โ claim-driven experiment roadmap from /experiment-plan
refine-logs/FINAL_PROPOSAL.md โ method description for implementation context
idea-stage/IDEA_CANDIDATES.md โ compact idea summary (preferred when COMPACT: true) (fall back to ./IDEA_CANDIDATES.md if not found)
idea-stage/IDEA_REPORT.md โ full brainstorm output (fall back to ./IDEA_REPORT.md if not found)
If none exist, ask the user what experiments to implement.
Workflow
Phase 1: Parse the Experiment Plan
Read EXPERIMENT_PLAN.md and extract:
Run order and milestones โ which experiments run first (sanity โ baseline โ main โ ablation โ polish)
For each experiment block:
Dataset / split / task
Compared systems and variants
Metrics to compute
Setup details (backbone, hyperparameters, seeds)
Success criterion
Priority (MUST-RUN vs NICE-TO-HAVE)
Compute budget โ total estimated GPU-hours
Method details from FINAL_PROPOSAL.md โ what exactly to implement
Present a brief summary:
๐ Experiment plan loaded:
- Milestones: [N] (sanity โ baseline โ main โ ablation)
- Must-run experiments: [N]
- Nice-to-have: [N]
- Estimated GPU-hours: [X]
Proceeding to implementation.
Research-contract fallback: if idea-stage/docs/research_contract.md does
not exist yet (idea selected outside /idea-discovery, or an older run),
create it now from templates/RESEARCH_CONTRACT_TEMPLATE.md using the selected
idea + claims from the experiment plan. Downstream /result-to-claim and
/ablation-planner read this file as the claims source, and session recovery
(docs/SESSION_RECOVERY_GUIDE.md) depends on it existing.
Phase 2: Implement Experiment Code
If BASE_REPO is set โ clone the repo first:
git clone <BASE_REPO> base_repo/
# Read the repo's README, understand its structure, find entry points# Implement experiments by modifying/extending this codebase
For each milestone (in order), write the experiment scripts:
Check existing code โ scan the project (or cloned base_repo/) for existing experiment scripts, model code, data loaders. Reuse as much as possible.
Implement missing pieces:
Training scripts with proper argparse (all hyperparameters configurable)
Evaluation scripts computing the specified metrics
Data loading / preprocessing if needed
Baseline implementations if not already present
Fixed random seeds for reproducibility
Results saved to JSON/CSV for later analysis
Proper logging (wandb if configured in CLAUDE.md)
Follow the plan's run order โ implement sanity-stage experiments first, then baselines, then main method, then ablations.
Self-review before deploying:
Are all hyperparameters from EXPERIMENT_PLAN.md reflected in argparse?
Is the random seed fixed and controllable?
Are results saved in a parseable format (JSON/CSV)?
Does the code match FINAL_PROPOSAL.md's method description?
Before deploying, send the experiment code to GPT-5.6-Sol xhigh for review:
mcp__codex__codex:
model: gpt-5.6-sol
config: {"model_reasoning_effort": "xhigh"}
prompt: |
Review the following experiment implementation for correctness.
## Experiment Plan:
[paste key sections from EXPERIMENT_PLAN.md]
## Method Description:
[paste from FINAL_PROPOSAL.md]
## Implementation:
[paste the experiment scripts]
Check for:
1. Does the code correctly implement the method described in the proposal?
2. Are all hyperparameters from the plan reflected in the code?
3. Are there any logic bugs (wrong loss function, incorrect data split, missing eval)?
4. Is the evaluation metric computed correctly?
5. **CRITICAL: Does evaluation use the dataset's actual ground truth labels โ NOT another model's output as ground truth?** This is a common and severe bug.
6. Any potential issues (OOM risk, numerical instability, missing seeds)?
For each issue found, specify: CRITICAL / MAJOR / MINOR and the exact fix.
On review results:
No CRITICAL issues โ proceed to Phase 3
CRITICAL issues found โ fix them, then re-submit for review (max 2 rounds)
Before deploying the full experiment suite, run the sanity-stage experiment:
/run-experiment [sanity experiment command]
Wait for completion. Verify:
Training loop runs without errors
Metrics are computed and saved correctly
GPU memory usage is within bounds
Output format matches expectations
If sanity fails โ auto-debug before giving up. Budget: up to 2 patch
attempts on the same failure, then up to 2 clean reimplements (4 total):
Read the error โ parse traceback, stderr, and log files. (The same
read-the-primary-artifact discipline applies to surprising REVIEWER verdicts:
see shared-references/review-tracing.md ยง Debugging With Traces.)
Diagnose โ classify the failure:
OOM โ reduce batch size or enable gradient checkpointing
ImportError โ install missing package
FileNotFoundError โ fix path or download data
CUDA error โ check GPU availability, reduce model size
NaN/divergence โ reduce learning rate, check data preprocessing
Fix and re-run โ apply the fix, re-run sanity
Attempt 2+ still failing? โ Call in Codex rescue (if Codex plugin installed):
Before the next retry, invoke /codex:rescue to get a second opinion on the root cause. Codex independently reads the code and error logs โ it may spot issues Claude missed (wrong tensor shapes, subtle import shadowing, config mismatches, etc.). Apply its suggested fix, then re-run.
If /codex:rescue is not available (plugin not installed), continue with Claude's own diagnosis
Both patch attempts failed on the same failure? โ Discard and reimplement cleanly
(up to 2 reimplements). Rewriting the failing script from EXPERIMENT_PLAN.md / the
research contract is a PEER move to another patch, not a last resort โ a third patch
on top of two wrong ones is usually worse than a clean rebuild. Delete ONLY the
attempt's own code/scaffolding (scripts this phase generated); the plan,
EXPERIMENT_TRACKER.md, user-authored project source, collected data, and results
are never deletable (see shared-references/external-cadence.md ยง Let a broken
attempt restart, not just patch).
Budget exhausted (2 patches + 2 reimplements), or two reimplements failed the SAME
way? โ stop, report the failure with all attempted fixes and error logs. Two clean
reimplements failing identically usually means the plan or the environment is wrong โ
say so explicitly in the report, because that (not the broken build itself) is what
needs the human. Do not proceed with broken code.
Never give up on the first failure. Most experiment crashes are fixable without human intervention.
Phase 4: Deploy Full Experiments
Deploy experiments following the plan's milestone order. Route by job count:
Small batch (โค5 jobs per milestone) โ use /run-experiment directly:
/run-experiment [experiment commands]
Large batch (โฅ10 jobs, multi-seed sweeps, or phase dependencies) โ use /experiment-queue for proper orchestration:
/experiment-queue [grid spec or manifest]
Auto-routing rule: if any milestone in EXPERIMENT_PLAN.md declares โฅ10 jobs (e.g., seeds: [42, 200, 201, ...] ร N: [64, 128, 256] ร n: [50K, 150K, 500K, 652K] = 36 jobs) or declares teacherโstudent phase dependencies, route that milestone to /experiment-queue. Otherwise use /run-experiment.
/experiment-queue adds: OOM-aware retry with backoff, stale-screen cleanup, wave-transition race prevention, phase dependency enforcement, crash-safe state persistence in queue_state.json. See skills/experiment-queue/SKILL.md for the manifest YAML format.
For each milestone:
Deploy experiments in parallel (up to MAX_PARALLEL_RUNS for /run-experiment, or max_parallel from manifest for /experiment-queue)
Use /monitor-experiment to track progress (reads from queue_state.json if /experiment-queue is active)
Collect results as experiments complete
๐ฆ Checkpoint (if AUTO_DEPLOY = false):
๐ง Code implementation complete. Ready to deploy:
Milestone 0 (sanity): [status โ passed/pending]
Milestone 1 (baseline): [N experiments, ~X GPU-hours]
Milestone 2 (main method): [N experiments, ~X GPU-hours]
Milestone 3 (ablations): [N experiments, ~X GPU-hours]
Total estimated: ~X GPU-hours on [N] GPUs
Deploy now? Or review the code first?
Phase 5: Collect Initial Results
As experiments complete:
Parse output files (JSON/CSV/logs) for key metrics
Training quality check โ if W&B data is available (CLAUDE.md has wandb: true and wandb_project), invoke /training-check to detect NaN, loss divergence, plateaus, or overfitting. If W&B is not configured, skip silently.
Update refine-logs/EXPERIMENT_TRACKER.md โ fill in Status and Notes columns
Check success criteria from EXPERIMENT_PLAN.md โ did each experiment meet its bar?
Write initial results summary:
# Initial Experiment Results**Date**: [today]
**Plan**: refine-logs/EXPERIMENT_PLAN.md
## Results by Milestone
### M0: Sanity โ PASSED
- [result]
### M1: Baselines
| Run | System | Key Metric | Status |
|-----|--------|-----------|--------|
| R001 | baseline_1 | X.XX | DONE |
### M2: Main Method
| Run | System | Key Metric | Status |
|-----|--------|-----------|--------|
| R003 | our_method | X.XX | DONE |
### M3: Ablations
...
## Summary
- [X/Y] must-run experiments completed
- Main result: [positive/negative/inconclusive]
- Ready for /auto-review-loop: [YES/NO]
## Next Step
โ /auto-review-loop "[topic]"
CRITICAL โ Evaluation must use dataset ground truth. When writing evaluation scripts, ALWAYS compare model predictions against the dataset's actual ground truth labels/targets โ NEVER use another model's output as ground truth. Double-check: (1) ground truth comes from the dataset split, not from a baseline/backbone model, (2) evaluation metrics are computed against the same ground truth for all methods, (3) if the task has official eval scripts, use those.
Follow the plan. Do not invent experiments not in EXPERIMENT_PLAN.md. If you think something is missing, note it but don't add it.
Sanity first. Never deploy a full suite without verifying the sanity stage passes.
Reuse existing code. Scan the project before writing new scripts. Extend, don't duplicate.
Save everything as JSON/CSV. The auto-review-loop needs parseable results, not just terminal output.
Update the tracker.EXPERIMENT_TRACKER.md should reflect real status after each run completes.
Don't wait forever. If an experiment exceeds 2x its estimated time, flag it and move on to the next milestone.
Budget awareness. Track GPU-hours against the plan's budget. Warn if approaching the limit.
Vast.ai lifecycle. If using vast.ai instances, destroy them after all experiments complete and results are downloaded. Running instances cost money every second โ don't leave them idle. Use /vast-gpu destroy or /vast-gpu destroy-all when done.
Modal lifecycle. If using gpu: modal, no cleanup is needed โ Modal auto-scales to zero after each run. But always show cost estimates before running and verify the spending limit is set at https://modal.com/settings (NEVER through CLI).
Composing with Other Skills
/idea-discovery "direction" โ Workflow 1: find + refine + plan
/experiment-bridge โ you are here (Workflow 1.5: implement + deploy)
/auto-review-loop "topic" โ Workflow 2: review + iterate
/paper-writing "NARRATIVE_REPORT.md" โ Workflow 3: write the paper
Or use /research-pipeline for the full end-to-end flow (includes this bridge).
