validate-training-pipeline

name	validate-training-pipeline
description	Training pipeline maintenance — validates experiment runner, training loop, training rig, and component integration for all notebooks.

Validate Training Pipeline

Mental Model

The training pipeline is the core infrastructure that turns notebooks into trained models. It consists of experiment_runner.py (lifecycle management), training_loop.py (epoch iteration), training_rig.py (model setup), and data_loading.py (dataset access). When any component breaks, training fails silently or produces invalid results. This skill ensures the pipeline is healthy, components integrate correctly, and notebooks use the pipeline properly.

Coverage

Documented: Experiment runner health, training loop validation, training rig integrity, component integration, notebook pipeline usage. Not yet documented: Training convergence detection, loss anomaly detection, gradient health checks. Last extended: 2026-06-24

---

What This Skill Checks

1. Experiment Runner Health

notebooks/components/experiment_runner.py manages the full training lifecycle.

Checks:

• Module imports successfully
• run_standard_experiment() function exists and is callable
• Function signature matches expected pattern
• Handles all notebook types (standard and experimental)

Verification command:

python -c "from notebooks.components.experiment_runner import run_standard_experiment; print('OK')"

2. Training Loop Validation

notebooks/components/training_loop.py handles epoch iteration.

Checks:

• Module imports successfully
• train() function exists and is callable
• Function accepts expected parameters (model, dataloader, optimizer, etc.)
• Returns expected outputs (loss, metrics)

Verification command:

python -c "from notebooks.components.training_loop import train; print('OK')"

3. Training Rig Integrity

notebooks/components/training_rig.py sets up the model architecture.

Checks:

• Module imports successfully
• TrainingRig class exists
• Class handles layer freezing, GPT2Block attachment, classifier head
• Class supports all disposition types (aversion, empathy, coherence, humility)

Verification command:

python -c "from notebooks.components.training_rig import TrainingRig; print('OK')"

4. Component Integration

All components should work together:

Integration flow:

1. Notebook calls run_standard_experiment()
2. Runner calls data_loading to get dataset
3. Runner calls TrainingRig to set up model
4. Runner calls training_loop.train() to train
5. Runner calls artifacts to save results

Checks:

• Components can be imported together without conflicts
• Function signatures are compatible
• Return types match expected inputs

5. Notebook Pipeline Usage

Each notebook should use the pipeline correctly:

Standard notebooks (01-04):

• Should only call run_standard_experiment()
• Should not contain training loops
• Should not contain model setup code

Experimental notebooks (01b, 01c, 01d):

• May have custom training loops
• Must still use device_setup.setup() for device init
• Must still use artifacts for saving results

Checks:

• Standard notebooks have no training code
• Experimental notebooks use shared components
• No duplicate code across notebooks

6. Error Handling

The pipeline should handle errors gracefully:

Checks:

• Training loop catches and logs exceptions
• Artifacts are saved even if training fails
• GPU memory is cleaned up on error
• MLflow runs are properly closed

7. Configuration Consistency

Notebook configurations should match pipeline expectations:

Checks:

• COMPONENT value is valid (aversion, empathy, coherence, humility)
• POSITIONS value is valid (0-11 for GPT-2 small)
• DATASET value matches a spec in data_loading.py
• OUTPUT_DIR follows naming convention

---

Anti-Patterns to Fix

1. Import failure — Component can't be imported
2. Signature mismatch — Function signature doesn't match usage
3. Missing error handling — Training crashes without saving artifacts
4. Duplicate code — Training logic implemented in notebook instead of components
5. Wrong configuration — Notebook config doesn't match pipeline expectations
6. Integration break — Components work individually but not together
7. Missing cleanup — GPU memory not freed on error

---

Actionable Steps

1. Import All Components

python -c "
from notebooks.components.experiment_runner import run_standard_experiment
from notebooks.components.training_loop import train
from notebooks.components.training_rig import TrainingRig
from notebooks.components.data_loading import load_dataset
from notebooks.components.artifacts import save_report
from notebooks.components.device_setup import setup
print('All components import OK')
"

2. Check Function Signatures

Read each component module and verify function signatures match expected usage.

3. Test Integration

Run a minimal integration test:

from notebooks.components.experiment_runner import run_standard_experiment
from notebooks.components.training_rig import TrainingRig
from notebooks.components.training_loop import train
# Verify components can work together

4. Check Notebook Usage

Read each notebook and verify it uses the pipeline correctly.

5. Fix Issues

• Fix import errors
• Update function signatures if needed
• Add missing error handling
• Extract duplicate code to components

---

Anti-Patterns to Fix (Specific)

Pattern	Where to Look	Fix
Import failure	Component module	Fix import path or missing dependency
Signature mismatch	Function definition	Update signature to match usage
Missing error handling	Training loop	Add try/finally blocks
Duplicate code	Notebooks	Extract to components/
Wrong config	Notebook constants	Update to valid values
Integration break	Component interaction	Fix compatibility

---

Known Violations

Check these specific locations first:

• notebooks/components/experiment_runner.py — Main entry point
• notebooks/components/training_loop.py — Epoch iteration
• notebooks/components/training_rig.py — Model setup
• All training notebooks — Pipeline usage

---

Coverage

Already clean:

• All components import successfully
• Standard notebooks use pipeline correctly

Still needs work:

• Integration testing (components working together)
• Error handling verification
• Configuration consistency checks

---

Verification

After fixing issues, verify:

# Check all components import:
python -c "
from notebooks.components.experiment_runner import run_standard_experiment
from notebooks.components.training_loop import train
from notebooks.components.training_rig import TrainingRig
from notebooks.components.data_loading import load_dataset
from notebooks.components.artifacts import save_report
from notebooks.components.device_setup import setup
print('All components OK')
"

# Check standard notebooks have no training code:
grep -l "for.*epoch\|backward()\|optimizer.step()" notebooks/0[1-4]*.py

# Check experimental notebooks use components:
grep -l "from notebooks.components" notebooks/01[bd]*.py notebooks/01c*.py

Report Format

STATUS: [no_work | fixed]

CHANGES:
- notebooks/components/training_loop.py: Added error handling for CUDA OOM
- notebooks/01b_teacher_distillation.py: Extracted duplicate training code to components

DETAILS:
{Detailed explanation of each change}