deploy-experiment

name	deploy-experiment
description	Deploy and run experiments for ML/DL training (local, remote, or Modal GPU) AND spatial data science / GIScience experiments (local, data-driven). Reads from output/refine-logs/EXPERIMENT_PLAN.md and output/refine-logs/FINAL_PROPOSAL.md, writes to output/experiment/. Use when user says "run experiment", "deploy experiment", "execute experiment plan", or needs to launch training / spatial analysis jobs.
argument-hint	["experiment-description-or-scope"]
allowed-tools	Bash(*), Read, Grep, Glob, Edit, Write, Agent

Deploy Experiment

Execute the experiment described in output/refine-logs/EXPERIMENT_PLAN.md (with context from output/refine-logs/FINAL_PROPOSAL.md) and write all results to output/experiment/.

Scope override (optional): $ARGUMENTS

---

Inputs

File	Purpose
output/refine-logs/EXPERIMENT_PLAN.md	Authoritative experiment plan: objectives, claims, run order, commands, success criteria, data needs
output/refine-logs/FINAL_PROPOSAL.md	Final refined proposal: problem anchor, method, expected contribution, constraints
CLAUDE.md	Environment config (gpu location, conda env, SSH, wandb, code_sync)

If either required input is missing, stop and tell the user which file to produce first (likely via /refine-research and /experiment-design).

Outputs (all under output/experiment/)

File	Content
EXPERIMENT_RESULT.md	Final results report: per-claim outcomes, tables, key numbers, figures referenced
EXPERIMENT_LOG.md	Chronological run log: commands executed, environment, timings, stdout/stderr pointers, failures, retries
data/	Intermediate data artifacts ready for visualization (CSV, Parquet, GeoJSON, GeoPackage, NetCDF, .npz, model predictions, metrics.json)
figures/ (optional)	Quick-look figures produced during the run (final paper figures come from paper-figure)
scripts/ (optional)	Any inline runner scripts created to execute the plan

Create the output/experiment/, output/experiment/data/, output/experiment/figures/, and output/experiment/scripts/ directories at the start if they do not exist.

---

Step 0: Mandatory Local GPU Availability Check (pilot AND full experiments)

This check is REQUIRED before classification, regardless of whether the run is a pilot, a full experiment, Track A, Track B, or mixed. The goal is to guarantee that when a local GPU is present, every ML / DL component runs on it — never silently on CPU.

Run the local GPU presence check first:

Local CUDA:

nvidia-smi --query-gpu=index,name,memory.used,memory.total --format=csv,noheader

Local Mac MPS (only if nvidia-smi is missing or returns zero rows):

python -c "import torch; print('MPS available:', torch.backends.mps.is_available())"

Decision rules (apply in order):

1. If nvidia-smi exits 0 and returns at least one GPU row → record LOCAL_GPU=cuda, capture the <gpu_id> and the chosen device for use in Step A5 and any pilot launch. Do NOT gate on idle memory; a desktop GPU with a display attached normally reports 0.7–1.5 GiB used.
2. Else if torch.backends.mps.is_available() returns True → record LOCAL_GPU=mps.
3. Else → record LOCAL_GPU=none. Only in this case may an ML / DL component fall back to CPU or to remote / Modal as defined in CLAUDE.md.

Write the result as the first lines of output/experiment/EXPERIMENT_LOG.md:

- LOCAL_GPU: cuda | mps | none
- Device chosen: <e.g. CUDA:0 / mps / cpu / remote-A100>
- GPU check command + output: <verbatim>

Hard rule: if LOCAL_GPU is cuda or mps, every ML / DL pilot AND every ML / DL full-experiment run launched by this skill MUST use that local GPU. Do not switch to remote / Modal / CPU just because the plan's default is set elsewhere — local GPU takes precedence whenever it is present. Record any deviation as a CONTRACT_VIOLATION.md with the user's explicit override.

If LOCAL_GPU=none and CLAUDE.md lists gpu: remote or gpu: modal, route ML / DL work there as documented in Step A1; otherwise stop and report to the user before proceeding.

This check applies equally to:

• Pilot experiments invoked from /generate-idea (Phase 5)
• Full experiments invoked from /full-pipeline Stage 2
• Any mixed GeoAI experiment whose Track A leg requires GPU

---

Step 0.1: Classify the Experiment

Read EXPERIMENT_PLAN.md and FINAL_PROPOSAL.md, then classify into ONE of the two tracks below. Record the choice at the top of EXPERIMENT_LOG.md.

Track	Signals	Route
A. ML / Deep Learning	Model training, fine-tuning, gradient descent, epochs, GPU/TPU required, checkpoints, benchmarks on splits, wandb-style tracking	→ Step A1
B. Spatial / GIScience	Spatial regression, ESDA (Moran's I, LISA, Getis-Ord), GWR/MGWR, spatial clustering, accessibility (2SFCA, isochrones), cartographic description, spatiotemporal analysis, network analysis, choropleth-driven claims, no GPU needed (mostly CPU + geospatial libs)	→ Step B1

Mixed experiments (e.g., GeoAI: deep learning applied to spatial data) run both tracks in sequence — Track A for model training/inference, Track B for downstream spatial evaluation.

Invoke supporting skills when needed:

• If the plan requires datasets not yet in data/raw/ → call data-download skill with the data requirements from the plan.
• For Track B (spatial analysis execution and diagnostics) → call spatial-analysis skill to run the analysis under guideline-driven decisions. This skill (deploy-experiment) orchestrates and records results; spatial-analysis is the executor for spatial methods.

---

Track A — ML / Deep Learning Experiments

A1. Detect Environment

From CLAUDE.md:

• Local GPU (gpu: local): local CUDA or Apple MPS
• Remote server (gpu: remote): SSH alias, conda env, code directory
• Modal (gpu: modal): use Modal serverless GPU app

A2. Pre-flight GPU Check

The local presence check from Step 0 has already determined LOCAL_GPU. Re-use that result — do not re-decide here. Only run remote / multi-GPU contention checks below.

Remote (SSH), only when LOCAL_GPU=none and CLAUDE.md says gpu: remote:

ssh <server> nvidia-smi --query-gpu=index,memory.used,memory.total --format=csv,noheader

Contention check (optional, only when sharing a multi-GPU box): pick a GPU where memory.free / memory.total > 0.7 to avoid colliding with another running job. On a single-GPU workstation, skip this and proceed once presence is confirmed in Step 0.

A3. Sync Code (Remote Only)

Read code_sync in CLAUDE.md (default rsync).

rsync:

rsync -avz --include='*.py' --exclude='*' <local_src>/ <server>:<remote_dst>/

git:

git add -A && git commit -m "sync: experiment deployment" && git push
ssh <server> "cd <remote_dst> && git pull"

A4. W&B Integration (only when wandb: true)

Skip entirely if not enabled. Otherwise, ensure scripts contain:

import wandb
wandb.init(project=WANDB_PROJECT, name=EXP_NAME, config={...})
wandb.log({"train/loss": loss, "train/lr": lr, "step": step})
wandb.log({"eval/loss": eval_loss, "eval/accuracy": acc})
wandb.finish()

Verify login: ssh <server> "wandb status".

A5. Launch

Remote (SSH + screen):

ssh <server> "screen -dmS <exp_name> bash -c '\
  eval \"\$(<conda_path>/conda shell.bash hook)\" && \
  conda activate <env> && \
  CUDA_VISIBLE_DEVICES=<gpu_id> python <script> <args> 2>&1 | tee <log_file>'"

Local:

CUDA_VISIBLE_DEVICES=<gpu_id> python <script> <args> 2>&1 | tee output/experiment/<exp_name>.log
# Mac MPS: omit CUDA_VISIBLE_DEVICES

Use run_in_background: true for long local jobs. Each experiment gets its own screen/process and one GPU.

A6. Collect Artifacts

After completion (detected via /monitor-experiment or screen ending):

rsync -avz -e "ssh -p <PORT>" root@<HOST>:/workspace/project/results/ output/experiment/data/
scp -P <PORT> root@<HOST>:/workspace/*.log output/experiment/

Place tabular metrics as metrics.csv / metrics.json, learned checkpoints' evaluation predictions as predictions.parquet (or .npz), and any training curves as curves.csv — all under output/experiment/data/ so they are visualization-ready.

---

Track B — Spatial / GIScience Experiments

These experiments are typically CPU-bound, run locally, and rely on PyGeoAI / GeoPandas / PySAL / libpysal / esda / spreg / mgwr / rasterio / xarray / networkx / scikit-mobility. GPU is irrelevant unless the plan specifies a deep-learning component (then also run Track A).

B1. Verify / Acquire Data

1. Read the Data section of EXPERIMENT_PLAN.md.
2. Check data/DATA_MANIFEST.md and data/raw/ for each required dataset.
3. For any missing dataset → invoke the data-download skill with the precise data need (geography, time range, spatial resolution, variables, format). Do NOT ad-hoc download; rely on the skill so provenance is recorded in DATA_MANIFEST.md.
4. After download, re-check presence and basic integrity (file exists, non-zero size, opens with the expected library).

B2. Pre-flight Environment Check

python - <<'PY'
import importlib, sys
required = ["geopandas","pysal","libpysal","esda","spreg","mgwr","rasterio","xarray","shapely","pyproj","numpy","pandas","matplotlib"]
missing = [p for p in required if importlib.util.find_spec(p) is None]
print("MISSING:", missing)
PY

If anything is missing, report to the user and stop — do NOT silently pip install into a shared env. Suggest the conda env from CLAUDE.md.

B3. Plan-to-Run Mapping

For every claim listed in EXPERIMENT_PLAN.md, identify:

Plan field	Concrete execution artifact
Claim / hypothesis	Section heading in EXPERIMENT_RESULT.md
Spatial unit	CRS, geometry file in output/experiment/data/
Method	Call into spatial-analysis skill (ESDA / regression / GWR / accessibility / etc.)
Diagnostics required	Only the diagnostics the claim actually needs (see spatial-analysis/SKILL.md §5.1 / §5.3). Do NOT run Moran's I residuals, MAUP sweeps, GWR, alternative-W sensitivity, or spatial CV by default — they apply only when the claim depends on them. When the plan is silent, follow spatial-analysis §5.3 and ask the user before adding heavyweight checks (and before omitting one that a strict GIScience reviewer would expect). Capture whatever diagnostics are run in data/diagnostics.json.
Success criterion	Pass/fail row in the results table

B4. Execute via the spatial-analysis Skill

Invoke the spatial-analysis skill per claim (or per grouped set of claims). Pass the research question verbatim from EXPERIMENT_PLAN.md and the resolved data paths. The skill will write analysis artifacts to output/spatial-analysis/ — this skill (deploy-experiment) is then responsible for:

1. Copying / linking the resulting tables and figures into output/experiment/data/ and output/experiment/figures/ with stable filenames.
2. Writing a short per-claim summary row (numbers, CI, p-values, effect sizes, diagnostics) to EXPERIMENT_RESULT.md.

Intermediate data files to persist — decide which artifacts are actually needed based on the claims, methods, and visualizations the plan implies. The list below is illustrative, not mandatory; produce only what is genuinely useful for downstream visualization, paper figures, or reproducibility, and skip the rest.

Examples (pick, rename, or add as appropriate):

• data/features.gpkg or .parquet — joined analytical table with geometries
• data/moran_results.csv — global + local indicators
• data/lisa_clusters.gpkg — LISA cluster/outlier labels
• data/regression_coefficients.csv — OLS / spatial lag / error / GWR coefficients
• data/gwr_local_estimates.gpkg — per-location coefficients, t-values, local R²
• data/predictions.parquet — out-of-sample predictions with geometry keys
• data/diagnostics.json — VIF, condition number, Moran's I on residuals, CV scores
• data/metrics.json — aggregate pass/fail per claim (machine-readable)

Always accompany data with metadata so the next step (figure generation, paper writing, review) can interpret it without re-reading code:

• data/README.md — index of every file under data/, one line each: filename → what it is, which claim it supports, which run produced it.
• Per-file sidecar: for each non-trivial artifact X.ext, write X.meta.json (or X.meta.yaml) containing at minimum:
  • description — one sentence on what the file holds
  • produced_by — script / skill invocation / run id in EXPERIMENT_LOG.md
  • source_inputs — upstream files used
  • schema — columns (name, dtype, unit, description) for tabular/vector files; variables + dims for raster/NetCDF
  • crs — EPSG code or WKT (for any geospatial file)
  • spatial_unit / temporal_range — when applicable
  • notes — caveats, filters applied, known issues

For simple CSVs, documenting columns in a header comment plus a one-line entry in data/README.md is sufficient; reserve full sidecar JSON for geospatial or multi-dimensional artifacts.

B5. For Mixed GeoAI Experiments

If the plan couples deep learning with spatial evaluation:

1. Run Track A to train/infer.
2. Export model predictions to output/experiment/data/predictions.parquet joined to spatial IDs.
3. Run Track B on those predictions (e.g., residual spatial autocorrelation, fairness across regions, spatially-stratified CV, LISA on error maps).

---

Human Checkpoint: Data Synthesis

Honor the HUMAN_CHECKPOINT flag in CLAUDE.md (default: true). When set, PAUSE and request explicit user approval before performing any of the synthesis actions below; resume only after the user replies. When false, append a one-line rationale to output/PROJ_NOTES.md and proceed. Never silently fabricate or substitute experiment data.

Trigger	Show before pausing
A required dataset is missing and the plan would be satisfied by simulated, imputed, or synthetically generated input data	Source(s) considered, exact synthesis recipe (sampler / generator / random seed), and which claims would rely on it
A run FAILED but the plan needs a value for that claim	The failure mode, the proposed substitute (e.g., last-checkpoint metric, k-fold mean, prior run, hand-set placeholder), and the impact on downstream claims
Metrics, predictions, or diagnostics need to be aggregated / re-weighted / re-scaled before being written to data/	Aggregation function, weights, denominator choice, and which raw artifacts are being collapsed
A Track B sidecar (*.meta.json) is being authored from inferred (not directly observed) schema, CRS, or units	Per-field provenance: observed vs inferred, source of inference
Mixed GeoAI Step B5 will join model predictions to spatial IDs via a key the plan does not specify	The join key proposed, fallback if matches < 100%, and risk of silent label leakage
EXPERIMENT_RESULT.md would mark a claim Pass using values that were not produced by an executed run in EXPERIMENT_LOG.md	The exact source of each number and why it is acceptable evidence

If the user declines a synthesis action, record the decision in EXPERIMENT_LOG.md under the affected run as Synthesis declined: <action> — <user reason> and either re-plan, mark the claim Blocked, or stop.

---

Step 5 (Both Tracks): Verify & Record

Remote verify:

ssh <server> "screen -ls"

Local verify: check process and, for Track A, GPU allocation.

Update output/experiment/EXPERIMENT_LOG.md continuously during the run. Use this structure:

# Experiment Log

- Started: <ISO timestamp>
- Track: A | B | A+B
- Plan: output/refine-logs/EXPERIMENT_PLAN.md
- Proposal: output/refine-logs/FINAL_PROPOSAL.md
- Environment: <conda env / python version / host>

## Run <N>: <short name>
- Command: `...`
- GPU / CPU: ...
- Start → End: ...
- Exit status: SUCCESS | FAILED | PARTIAL
- Stdout/stderr: <path>
- Artifacts produced: <paths under output/experiment/data or figures>
- Notes: <anything surprising>

---

Step 6 (Both Tracks): Write EXPERIMENT_RESULT.md

After all runs complete (or if EXPERIMENT_PLAN.md marks the run as terminal), compose output/experiment/EXPERIMENT_RESULT.md with this structure:

# Experiment Result

> Plan: output/refine-logs/EXPERIMENT_PLAN.md
> Proposal: output/refine-logs/FINAL_PROPOSAL.md
> Track: A | B | A+B
> Date: <ISO>

## 1. Summary
<3–6 sentences: what was run, what was found, whether the headline claim holds>

## 2. Per-Claim Outcomes
| Claim | Success criterion | Observed | Pass/Fail |
|---|---|---|---|
| C1 ... | ... | ... | ✅ / ❌ |

## 3. Key Numbers and Tables
<inline the most important metrics; reference files under output/experiment/data/>

## 4. Diagnostics
<Track A: loss curves, generalization gap, ablations>
<Track B: **only the diagnostics the claim required** — pick from Moran's I on residuals, VIF, spatial CV, GWR local R², LISA significance, etc. Explicitly list any geospatial check that was considered but skipped, with the one-line reason (e.g., "MAUP sensitivity skipped: unit of analysis is fixed sensor location, not aggregated"). See `spatial-analysis/SKILL.md` §5 for the trigger conditions.>

## 5. Intermediate Artifacts (for visualization)
<list files under output/experiment/data/ with one-line descriptions so paper-figure can pick them up>

## 6. Known Limitations / Failures
<anything marked FAILED or PARTIAL in EXPERIMENT_LOG.md>

Append a one-line entry to output/EXPERIMENT_LOG.md (the repo-level log listed in CLAUDE.md) pointing to the new output/experiment/EXPERIMENT_RESULT.md.

---

Key Rules

• ALWAYS read EXPERIMENT_PLAN.md and FINAL_PROPOSAL.md before any execution; never improvise the experiment.
• ALWAYS run the Step 0 local GPU presence check before any pilot or full experiment. If LOCAL_GPU is cuda or mps, every ML/DL run MUST execute on the local GPU; fall back to remote / Modal / CPU only when LOCAL_GPU=none.
• Route data acquisition through the data-download skill — preserves provenance.
• Route spatial execution through the spatial-analysis skill — preserves generator-evaluator separation.
• For Track A: GPU is already verified in Step 0; one experiment = one screen/process = one GPU.
• Use tee or explicit log files so EXPERIMENT_LOG.md can reference them.
• Persist intermediate data under output/experiment/data/ with documented schemas so figures are reproducible.
• Never fabricate numbers. If a run fails, record FAILED and do NOT invent a plausible result.
• Run deployment commands with run_in_background: true when long-running to keep the conversation responsive.
• Report back: track chosen, runs launched, where logs/artifacts are, estimated time, next step.

---

CLAUDE.md Example

## Remote Server
- gpu: remote
- SSH: `ssh my-gpu-server`
- GPU: 4x A100 (80GB each)
- Conda: `eval "$(/opt/conda/bin/conda shell.bash hook)" && conda activate research`
- Code dir: `/home/user/experiments/`
- code_sync: rsync          # or "git"
- wandb: false
- wandb_project: my-project
- wandb_entity: my-team

## Local Environment
- gpu: local
- Mac MPS / Linux CUDA
- Conda env: `ml` (Python 3.10 + PyTorch)

## Spatial Environment
- conda env: `geo` (geopandas, pysal, mgwr, rasterio, xarray)
- CRS default: EPSG:5070 (CONUS) / EPSG:4326 (global)

W&B: run wandb login on the server once (or set WANDB_API_KEY). Dashboard: https://wandb.ai/<entity>/<project>.