validate-notebooks

name	validate-notebooks
description	Validates standard/experimental notebook conventions, shared component usage, and lifecycle hooks. Provides grep-based checks and targeted auto-fix guidance.

Validate Notebook Conventions

Mental Model

Notebooks are the experiment interface. Two tiers exist:

• Standard notebooks (01–04): Thin configuration wrappers around run_standard_experiment(). They contain almost no logic.
• Experimental notebooks (01b, 01c, 01d, ...): Custom training approaches with more freedom, but mandatory lifecycle hooks.

The golden rule: if you find yourself writing the same code in two notebooks, it belongs in components/.

---

What to Check

Standard Notebooks (01, 02, 03, 04)

These MUST contain ONLY:

1. A markdown header (# %% [markdown])
2. A configuration block (constants: COMPONENT, POSITIONS, MAX_EXAMPLES, etc.)
3. A single call to run_standard_experiment()

That's it. No setup, no teardown, no training loops, no artifact saving. The runner handles all of that.

Violation patterns and detection:

Violation	Grep pattern	Expected for standard
Training loop	for.*epoch	NO matches
Backward pass	\.backward\(\)	NO matches
Optimizer step	optimizer\.step|\.step\(\)	NO matches
Manual torch.save	torch\.save	NO matches
Manual report save	open.*report\.json|save_report	NO matches
Manual device setup	torch\.device|cuda.*is_available	NO matches
Manual seed	torch\.manual_seed|random\.seed	NO matches

Experimental Notebooks (01b, 01c, 01d, ...)

These MAY contain custom training loops, losses, and artifact generation. But they MUST:

1. Use device_setup.setup() for device/seed init (or manual equivalent with identical structure)
2. Call artifacts.setup_run_logging(output_dir) immediately after creating the output dir
3. Call artifacts.teardown_run_logging() when done (use try/finally)
4. Save report.json via artifacts.save_report() — always, even if the experiment fails
5. Use artifacts.save_pca_plot() / save_confusion_matrix() for standard plots — don't reimplement
6. Clean up rigs with del rig; torch.cuda.empty_cache()
7. Stop the VRAM monitor at the end: env["vram_monitor"].stop()

Violation patterns and detection:

Violation	Grep pattern	Fix
Missing teardown	No teardown_run_logging in file	Wrap main body in try/finally
Missing report save	No save_report in file	Add artifacts.save_report(...) call
Reimplemented PCA	plt.figure + PCA( without save_pca_plot	Use artifacts.save_pca_plot()
Reimplemented confusion matrix	confusion_matrix + plt.imshow without save_confusion_matrix	Use artifacts.save_confusion_matrix()
Missing CUDA cleanup	No torch.cuda.empty_cache in file	Add del <model>; torch.cuda.empty_cache()
Missing VRAM stop	No vram_monitor.stop in file	Add env["vram_monitor"].stop()

Shared Component Usage

All notebooks should import from notebooks/components/ rather than reimplementing:

Functionality	Use this	Don't reimplement
Device init	device_setup.setup()	Manual torch.device() + seed
Data loading	data_loading module	Custom load_dataset calls
Training loop	training_loop.train()	Manual epoch/batch loops in standard notebooks
Artifact saving	artifacts module	Direct torch.save / json.dump
MLflow logging	mlflow_tracking module	Raw mlflow.log_param calls
PCA plots	artifacts.save_pca_plot()	Manual matplotlib PCA code
Confusion matrices	artifacts.save_confusion_matrix()	Manual plt.imshow + confusion_matrix

---

Validation Commands

Run these to check notebook health. Each command returns files that violate the rule.

# --- Standard notebook checks (should return empty) ---

# Check standard notebooks have run_standard_experiment:
grep -rL "run_standard_experiment" notebooks/0[1-4]_*.py

# Check standard notebooks do NOT have training loops:
grep -l "for.*epoch" notebooks/0[1-4]_*.py 2>/dev/null

# Check standard notebooks do NOT have manual saves:
grep -l "torch\.save\|open.*report" notebooks/0[1-4]_*.py 2>/dev/null

# Check standard notebooks do NOT have device setup boilerplate:
grep -l "torch\.device\|cuda.*is_available" notebooks/0[1-4]_*.py 2>/dev/null

# --- Experimental notebook checks (should all find matches) ---

# Check experimental notebooks have teardown:
grep -rL "teardown_run_logging" notebooks/01[bcdefgh]*.py 2>/dev/null

# Check experimental notebooks have report save:
grep -rL "save_report\|report\.json" notebooks/01[bcdefgh]*.py 2>/dev/null

# Check experimental notebooks have CUDA cleanup:
grep -rL "torch\.cuda\.empty_cache" notebooks/01[bcdefgh]*.py 2>/dev/null

# Check experimental notebooks have VRAM monitor stop:
grep -rL "vram_monitor.*stop\|\.stop()" notebooks/01[bcdefgh]*.py 2>/dev/null

Cross-notebook duplication scan

For detecting code that should be extracted to components/:

# Find similar function/class definitions across notebooks:
grep -n "^class \|^def " notebooks/01[bcdefgh]*.py | sort -t: -k2

# Find repeated import blocks (compare first 40 lines of each notebook):
for f in notebooks/0[1-4]*.py notebooks/01[bcdefgh]*.py; do
  echo "=== $f ==="
  head -40 "$f" | grep "^import \|^from "
done

For deeper structural duplication, use jscpd:

npx jscpd notebooks/ --pattern "**/*.py" --min-tokens 50 --min-lines 5

---

Anti-Patterns and Fixes

Standard notebook has training loop

Symptom: for epoch in range(...) with backward() and optimizer.step() in a 01–04 file. Fix: Replace the entire training logic with a call to run_standard_experiment(). Keep only the config block and the runner call. See notebooks/01_train_aversion_layer.py for the canonical structure.

Standard notebook has manual device setup

Symptom: torch.device("cuda"...) or torch.manual_seed() in a 01–04 file. Fix: Delete it. The runner calls device_setup.setup() internally.

Experimental notebook missing teardown

Symptom: No teardown_run_logging() call, or it's outside a try/finally. Fix: Wrap the main body in:

artifacts.setup_run_logging(output_dir)
try:
    # ... main experiment code ...
    artifacts.save_report(report, output_dir)
finally:
    artifacts.teardown_run_logging()

Experimental notebook reimplements PCA plot

Symptom: Manual PCA(n_components=2) + plt.scatter() code that produces a before/after hidden-state visualization. Fix: Replace with artifacts.save_pca_plot(hidden_states_before, hidden_states_after, labels, output_dir / "pca_comparison.png").

Experimental notebook reimplements confusion matrix

Symptom: Manual confusion_matrix() + plt.imshow() code. Fix: Replace with artifacts.save_confusion_matrix(labels, predictions, output_dir / "confusion_matrix.png").

Missing CUDA cleanup after model deletion

Symptom: del model or del rig without a following torch.cuda.empty_cache(). Fix: Always pair model deletion with cache cleanup: del rig; torch.cuda.empty_cache().

Missing VRAM monitor stop

Symptom: No vram_monitor.stop() at the end of the notebook. Fix: Add vram_monitor.stop() as the final executable line.

---

Verification Checklist

After making fixes, verify with these commands:

# 1. Standard notebooks are clean wrappers
echo "=== Standard notebooks with training loops (should be empty) ==="
grep -l "for.*epoch" notebooks/0[1-4]_*.py 2>/dev/null || echo "  (none)"

echo "=== Standard notebooks with manual device setup (should be empty) ==="
grep -l "torch\.device" notebooks/0[1-4]_*.py 2>/dev/null || echo "  (none)"

echo "=== Standard notebooks with run_standard_experiment (should all match) ==="
grep -l "run_standard_experiment" notebooks/0[1-4]_*.py

# 2. Experimental notebooks have lifecycle hooks
echo "=== Experimental notebooks missing teardown (should be empty) ==="
grep -rL "teardown_run_logging" notebooks/01[bcdefgh]*.py 2>/dev/null || echo "  (none)"

echo "=== Experimental notebooks missing report save (should be empty) ==="
grep -rL "save_report" notebooks/01[bcdefgh]*.py 2>/dev/null || echo "  (none)"

echo "=== Experimental notebooks missing CUDA cleanup (should be empty) ==="
grep -rL "torch\.cuda\.empty_cache" notebooks/01[bcdefgh]*.py 2>/dev/null || echo "  (none)"

echo "=== Experimental notebooks missing VRAM stop (should be empty) ==="
grep -rL "vram_monitor.*stop" notebooks/01[bcdefgh]*.py 2>/dev/null || echo "  (none)"

---

Report Format

STATUS: [no_work | fixed | violations_found]

CHECKS RUN:
- Standard notebook structure: [pass/fail]
- Standard notebook purity (no training loops): [pass/fail]
- Experimental lifecycle hooks: [pass/fail]
- Shared component usage: [pass/fail]
- Cross-notebook duplication: [pass/fail with details]

CHANGES (if any):
- notebooks/<file>: <what was fixed>

REMAINING (if violations_found):
- notebooks/<file>: <violation description>

---

Reference: Canonical Structures

Standard notebook (canonical)

# %% [markdown]
# # Train <Disposition> Layer
# <description>

# %% Imports
from components import data_loading
from components.experiment_runner import run_standard_experiment

# %% ===== CONFIGURATION =====
COMPONENT = "<disposition>"
POSITIONS = [2]
MAX_EXAMPLES = 200000
BATCH_SIZE = 32
EPOCHS = 6
LEARNING_RATE = 2e-4
WEIGHT_DECAY = 0.01
SEED = 42
DATASET_SPEC = data_loading.AVERSION_30K

# %% Run
run_standard_experiment(
    notebook_id="<id>",
    component=COMPONENT,
    dataset_spec=DATASET_SPEC,
    positions=POSITIONS,
    max_examples=MAX_EXAMPLES,
    batch_size=BATCH_SIZE,
    epochs=EPOCHS,
    learning_rate=LEARNING_RATE,
    weight_decay=WEIGHT_DECAY,
    seed=SEED,
)

Experimental notebook lifecycle (minimal required structure)

# %% Imports
import ...
from components import artifacts, device_setup

# %% Setup
env = device_setup.setup(seed=42)
output_dir = Path("outputs/<experiment>_<run_id>")
output_dir.mkdir(parents=True, exist_ok=True)
artifacts.setup_run_logging(output_dir)

try:
    # %% Main experiment
    <custom training code>

    # %% Save report
    artifacts.save_report(report, output_dir)

finally:
    artifacts.teardown_run_logging()

# %% Cleanup
del <model>
torch.cuda.empty_cache()
env["vram_monitor"].stop()