training-check

name	training-check
description	Monitors running spatial experiments. Checks output files, log files, and process status. Categorizes results as OK, STALLED, FAILED, or COMPLETE. Fires alerts by appending to output/PROJ_NOTES.md. Run every 15 minutes during Stage 3 of research-pipeline.
tools	Bash, Read, Write

Skill: training-check

You monitor spatial experiment execution and detect problems early to avoid wasting compute time.

---

Phase 1: Check Active Experiments

Read output/EXPERIMENT_LOG.md for experiments with status RUNNING or PENDING.

For each running experiment:

1. Check if output file was recently modified:

python -c "import os, time; f='[output_path]'; age=(time.time()-os.path.getmtime(f))/60; print(f'Modified {age:.1f} min ago')"

2. Check last few lines of log file for errors:

tail -20 [log_path]

3. Check for error keywords: Error, Traceback, NaN, inf, MemoryError, Killed
4. Check for stall: no file modification in > 30 min despite RUNNING status

---

Phase 2: Classify Status

Signal	Classification	Action
Output file updated in last 15 min, no errors	OK	Continue monitoring
Log contains "NaN" or diverging loss	CLEARLY BAD	Terminate, log failure
Output file not modified in > 30 min	STALLED	Investigate process
Results file written, contains valid metrics	COMPLETE	Update EXPERIMENT_LOG
MemoryError in log	MEMORY FAIL	Retry with smaller sample
Process not running, no output file	DEAD	Re-queue if < 2 retries

---

Phase 3: Actions

COMPLETE: Update output/EXPERIMENT_LOG.md status to SUCCESS. Append finding to output/PROJ_NOTES.md. CLEARLY BAD: Kill job if possible. Mark as FAILED in log. Try to fix the issue and retry once. STALLED: Check if process is still alive. If dead: re-queue. If alive: wait 15 more min.

---

Phase 3.5: Human Checkpoint — Data Synthesis

Honor the HUMAN_CHECKPOINT flag in CLAUDE.md (default: true). This is a monitoring skill, but a few automatic recovery actions silently produce or substitute experiment data. When HUMAN_CHECKPOINT is true, PAUSE and request explicit user approval before any of the following; when false, log the action to output/PROJ_NOTES.md with rationale and proceed.

Trigger	Show before pausing
MEMORY FAIL → retry with smaller sample: about to subsample, downsample, or otherwise reduce the dataset to make the run fit	Original N, proposed N, sampling rule (random / stratified / spatial), seed, and the claim affected by the reduced sample
CLEARLY BAD → fix and retry: about to modify the run (reduce learning rate, drop a feature, change loss, swap optimizer) and re-launch	Diagnosed cause, the exact code/config change, and confirmation that the new run is still valid evidence for the original claim
DEAD → re-queue: about to re-launch a run whose previous outputs were partial, with carry-over from a checkpoint or warm-start	Which artifacts will be reused, which will be regenerated, and the risk of mixing pre-/post-fix data in the same result file
COMPLETE → mark SUCCESS: about to write a SUCCESS row using metrics that were imputed / interpolated across missing log lines, or backfilled from a different run	Which numbers were imputed, source, and whether the run should be re-executed instead

Do not mark a run SUCCESS in EXPERIMENT_LOG.md based on synthesized metrics. If the user approves, append Synthesis approved: <action> — <user reason> — <date> to the run's notes.

---

Phase 4: Progress Report

Output to stdout:

Training Check — <timestamp>
Active experiments: N
  COMPLETE: N
  OK (running): N
  STALLED: N
  FAILED: N

[List any failures or alerts]

Append alerts to output/PROJ_NOTES.md: [ALERT] Experiment <name> failed: <reason>