| name | training-check |
| description | Monitors running spatial experiments. Checks output files, log files, and process status. Categorizes results as OK, STALLED, FAILED, or COMPLETE. Fires alerts by appending to output/PROJ_NOTES.md. Run every 15 minutes during Stage 3 of research-pipeline. |
| tools | Bash, Read, Write |
Skill: training-check
You monitor spatial experiment execution and detect problems early to avoid wasting compute time.
Phase 1: Check Active Experiments
Read output/EXPERIMENT_LOG.md for experiments with status RUNNING or PENDING.
For each running experiment:
- Check if output file was recently modified:
python -c "import os, time; f='[output_path]'; age=(time.time()-os.path.getmtime(f))/60; print(f'Modified {age:.1f} min ago')"
- Check last few lines of log file for errors:
tail -20 [log_path]
- Check for error keywords:
Error, Traceback, NaN, inf, MemoryError, Killed
- Check for stall: no file modification in > 30 min despite RUNNING status
Phase 2: Classify Status
| Signal | Classification | Action |
|---|
| Output file updated in last 15 min, no errors | OK | Continue monitoring |
| Log contains "NaN" or diverging loss | CLEARLY BAD | Terminate, log failure |
| Output file not modified in > 30 min | STALLED | Investigate process |
| Results file written, contains valid metrics | COMPLETE | Update EXPERIMENT_LOG |
| MemoryError in log | MEMORY FAIL | Retry with smaller sample |
| Process not running, no output file | DEAD | Re-queue if < 2 retries |
Phase 3: Actions
COMPLETE: Update output/EXPERIMENT_LOG.md status to SUCCESS. Append finding to output/PROJ_NOTES.md.
CLEARLY BAD: Kill job if possible. Mark as FAILED in log. Try to fix the issue and retry once.
STALLED: Check if process is still alive. If dead: re-queue. If alive: wait 15 more min.
Phase 3.5: Human Checkpoint — Data Synthesis
Honor the HUMAN_CHECKPOINT flag in CLAUDE.md (default: true). This is a monitoring skill, but a few automatic recovery actions silently produce or substitute experiment data. When HUMAN_CHECKPOINT is true, PAUSE and request explicit user approval before any of the following; when false, log the action to output/PROJ_NOTES.md with rationale and proceed.
| Trigger | Show before pausing |
|---|
| MEMORY FAIL → retry with smaller sample: about to subsample, downsample, or otherwise reduce the dataset to make the run fit | Original N, proposed N, sampling rule (random / stratified / spatial), seed, and the claim affected by the reduced sample |
| CLEARLY BAD → fix and retry: about to modify the run (reduce learning rate, drop a feature, change loss, swap optimizer) and re-launch | Diagnosed cause, the exact code/config change, and confirmation that the new run is still valid evidence for the original claim |
| DEAD → re-queue: about to re-launch a run whose previous outputs were partial, with carry-over from a checkpoint or warm-start | Which artifacts will be reused, which will be regenerated, and the risk of mixing pre-/post-fix data in the same result file |
| COMPLETE → mark SUCCESS: about to write a SUCCESS row using metrics that were imputed / interpolated across missing log lines, or backfilled from a different run | Which numbers were imputed, source, and whether the run should be re-executed instead |
Do not mark a run SUCCESS in EXPERIMENT_LOG.md based on synthesized metrics. If the user approves, append Synthesis approved: <action> — <user reason> — <date> to the run's notes.
Phase 4: Progress Report
Output to stdout:
Training Check — <timestamp>
Active experiments: N
COMPLETE: N
OK (running): N
STALLED: N
FAILED: N
[List any failures or alerts]
Append alerts to output/PROJ_NOTES.md: [ALERT] Experiment <name> failed: <reason>
