validate-model-artifacts

name	validate-model-artifacts
description	Model checkpoint maintenance — validates .pt file integrity, metadata structure, version tracking, and storage health for trained disposition layers.

Validate Model Artifacts

Mental Model

Model artifacts (.pt files) are the trained disposition layers — the output of expensive GPU training runs. Each checkpoint contains both weights and metadata that must be valid for the model to load correctly. Corrupted checkpoints waste GPU hours and break downstream experiments. This skill ensures every checkpoint is loadable, well-formed, and properly tracked.

Coverage

Documented: Checkpoint integrity, metadata structure, version tracking, storage health, orphan detection. Not yet documented: Checkpoint size trend analysis, weight corruption detection, cross-checkpoint comparison. Last extended: 2026-06-24

---

What This Skill Checks

1. Checkpoint Integrity

Every .pt file in resources/models/ and resources/trained_layers/ must be loadable.

Verification command:

python -c "
import torch
ckpt = torch.load('path/to/checkpoint.pt', map_location='cpu', weights_only=False)
print('OK' if 'layer_state_dict' in ckpt and 'metadata' in ckpt else 'MISSING KEYS')
"

Checks:

• File is valid PyTorch serialization (not corrupted)
• Contains layer_state_dict key
• Contains metadata key
• Weights load without error

2. Metadata Structure

Every checkpoint must have complete metadata:

Field	Required	Type	Description
component	Yes	str	Disposition type (aversion, empathy, coherence, humility)
insert_position	Yes	int	Transformer layer position (0-11 for GPT-2 small)
training_date	Yes	str	ISO 8601 date
dataset	Yes	str	Training dataset name
epochs	Yes	int	Training epochs completed
final_loss	Yes	float	Final training loss
model_version	No	str	GPT-2 version used

3. Version Tracking

Checkpoints should follow a naming convention:

• Pattern: {component}_pos{position}_{date}.pt
• Example: aversion_pos2_20260624.pt

Checks:

• Filename matches expected pattern
• No duplicate checkpoints for same component+position (unless intentional versions)
• Metadata version matches filename version

4. Storage Health

Checks:

• Checkpoint files are not empty (0 bytes)
• Checkpoint files are reasonable size (GPT-2 Block ~5MB, not 500MB or 5KB)
• No orphaned checkpoint files (no matching experiment run)
• No orphaned experiment runs (no matching checkpoint file)

5. Cross-Experiment Consistency

Check that checkpoint metadata matches the experiment that produced it:

• component matches notebook name (01→aversion, 02→empathy, 03→coherence, 04→humility)
• insert_position matches the notebook's POSITIONS config
• dataset matches the notebook's data configuration

---

Anti-Patterns to Fix

1. Corrupted checkpoint — File exists but can't be loaded with torch.load()
2. Missing metadata — Checkpoint has weights but no metadata dict
3. Missing required fields — Metadata missing component, insert_position, or training_date
4. Wrong filename convention — Checkpoint named randomly instead of {component}_pos{position}_{date}.pt
5. Empty checkpoint — File is 0 bytes (likely interrupted save)
6. Orphaned checkpoint — No matching experiment run in MLflow
7. Size anomaly — Checkpoint significantly larger/smaller than expected

---

Actionable Steps

For each checkpoint file:

1. List all .pt files in resources/models/ and resources/trained_layers/
2. Try loading each with torch.load():

   import torch
   try:
       ckpt = torch.load('path.pt', map_location='cpu', weights_only=False)
       print('OK')
   except Exception as e:
       print(f'FAIL: {e}')
   

3. Check metadata structure against the required fields table
4. Verify filename convention matches {component}_pos{position}_{date}.pt
5. Check file size — GPT-2 Block checkpoint should be ~5MB (not 5KB or 500MB)
6. Cross-reference with MLflow — Check if experiment run exists for this checkpoint
7. Report issues and fix what can be auto-fixed (rename files, add missing metadata fields)

---

Anti-Patterns to Fix (Specific)

Pattern	Where to Look	Fix
Corrupted file	torch.load() fails	Re-run training or restore from backup
Missing metadata	ckpt.keys()	Add metadata dict with required fields
Bad filename	resources/models/	Rename to {component}_pos{position}_{date}.pt
Empty file	os.path.getsize() == 0	Delete and re-run training
Orphaned checkpoint	No matching MLflow run	Log the run manually or delete checkpoint

---

Known Violations

Check these specific locations first:

• resources/models/*.pt — All model checkpoints
• resources/trained_layers/*.pt — All trained layer checkpoints
• Any checkpoint created recently may not follow naming convention

---

Coverage

Already clean:

• Most existing checkpoints have valid metadata
• Most follow naming conventions

Still needs work:

• Orphan detection (cross-reference with MLflow)
• Size anomaly detection
• Checkpoint deduplication

---

Verification

After fixing issues, verify:

# Check all checkpoints load:
python -c "
import torch, glob
for f in glob.glob('resources/models/*.pt') + glob.glob('resources/trained_layers/*.pt'):
    try:
        ckpt = torch.load(f, map_location='cpu', weights_only=False)
        assert 'layer_state_dict' in ckpt, f'{f}: missing layer_state_dict'
        assert 'metadata' in ckpt, f'{f}: missing metadata'
        print(f'OK: {f}')
    except Exception as e:
        print(f'FAIL: {f}: {e}')
"

# Check naming convention:
ls resources/models/*.pt resources/trained_layers/*.pt | grep -v "_pos[0-9]*_"

Report Format

STATUS: [no_work | fixed]

CHANGES:
- resources/models/aversion_pos2.pt: Renamed to aversion_pos2_20260624.pt
- resources/trained_layers/empathy_pos5.pt: Added missing metadata fields

DETAILS:
{Detailed explanation of each change}