training-monitor

name	training-monitor
description	Monitor and analyze Esperanto training runs. Checks GPU usage, training metrics, W&B logs, and checkpoint status. Use when user asks about training progress, GPU status, or experiment monitoring.
allowed-tools	Read, Glob, Grep, Bash

Training Monitor Skill

Purpose

Help users monitor active or completed Esperanto training experiments.

Instructions

When monitoring training:

1. Check GPU Status

   nvidia-smi --query-gpu=name,memory.used,memory.total,utilization.gpu --format=csv
   

2. Find Active Experiments

   • Look for running Python processes: ps aux | grep train.py
   • Check recent log files in logs/

3. Analyze Experiment Results

   • Find checkpoints in experiments/results/ or checkpoints/
   • Read training_state.pt for step count and metrics
   • Check W&B run folders in wandb/

4. Key Metrics to Report

   • Success rate (rollout/success_rate)
   • Message norms (communication/message_norms)
   • Policy loss trend
   • Topographic similarity
   • Compositionality score

Example Queries

• "How is training going?" → Check GPU, find active runs, report metrics
• "What's the GPU memory usage?" → Run nvidia-smi
• "Show me the latest experiment results" → Find most recent checkpoint
• "Is the model learning?" → Check success rate and loss trends

Output Format

When reporting, use this structure:

## Training Status

**GPU**: [name] | [memory used/total] | [utilization]%

**Active Runs**: [list or "none"]

**Latest Checkpoint**: [path]
- Step: X
- Success Rate: X%
- Message Norms: X

**Trends**: [improving/stable/declining]