validate-mlflow

name	validate-mlflow
description	MLflow experiment tracking maintenance — validates run metadata, experiment organization, artifact consistency, and SQLite backend health.

Validate MLflow Tracking

Mental Model

MLflow is the project's experiment memory. Every training run logs parameters, metrics, and artifacts to notebooks/mlruns/. When MLflow tracking is inconsistent, you can't reproduce experiments, compare results, or trace which model came from which run. This skill ensures the MLflow database is healthy, runs are well-organized, and artifacts are properly logged.

Coverage

Documented: Run metadata validation, experiment organization, artifact consistency, SQLite backend health, orphan detection. Not yet documented: Metric trend analysis, run comparison validation, MLflow UI accessibility. Last extended: 2026-06-24

---

What This Skill Checks

1. SQLite Backend Health

The MLflow tracking backend uses SQLite at notebooks/mlruns/mlflow.db.

Verification command:

python -c "
import sqlite3
conn = sqlite3.connect('notebooks/mlruns/mlflow.db')
cursor = conn.execute('SELECT COUNT(*) FROM runs')
print(f'Runs in database: {cursor.fetchone()[0]}')
conn.close()
print('Backend healthy')
"

Checks:

• Database file exists and is readable
• Database is not corrupted (can execute queries)
• Tables exist (runs, experiments, metrics, params, tags)
• No locked database files

2. Experiment Organization

Every experiment should be properly organized:

Checks:

• Experiments have meaningful names (not just IDs)
• Runs are grouped under appropriate experiments
• No orphaned runs (run exists but experiment doesn't)
• Run names follow a pattern (e.g., aversion_pos2_20260624)

3. Run Metadata Validation

Every run should have complete metadata:

Field	Required	Description
run_id	Yes	Unique run identifier
experiment_id	Yes	Parent experiment
status	Yes	FINISHED, FAILED, or RUNNING
start_time	Yes	Run start timestamp
end_time	Yes (if finished)	Run end timestamp

Checks:

• All runs have required fields
• FINISHED runs have end_time
• No runs stuck in RUNNING state (likely interrupted)
• Run durations are reasonable (not 0 seconds or 100 hours)

4. Parameter and Metric Logging

Every training run should log key parameters and metrics:

Expected parameters:

• component (aversion, empathy, coherence, humility)
• insert_position (layer position)
• learning_rate
• batch_size
• epochs
• dataset

Expected metrics:

• train_loss (per epoch or final)
• val_loss (if validation used)
• accuracy (if classification)

Checks:

• Required parameters are logged
• Required metrics are logged
• Parameter values match the notebook that created the run
• Metric values are reasonable (loss > 0, accuracy between 0-1)

5. Artifact Consistency

Every run should have consistent artifacts:

Checks:

• report.json exists for finished runs
• Model weights (.pt files) are logged if training completed
• Plots (PCA, confusion matrix) are logged if applicable
• Artifact paths in MLflow match actual files on disk

6. Orphan Detection

Orphaned runs: Run exists in MLflow but no matching checkpoint file Orphaned checkpoints: Checkpoint file exists but no matching MLflow run

Cross-reference:

• MLflow runs → resources/models/*.pt and resources/trained_layers/*.pt
• Checkpoint files → MLflow runs

---

Anti-Patterns to Fix

1. Corrupted database — mlflow.db exists but can't be queried
2. Orphaned runs — Run exists but experiment doesn't
3. Missing parameters — Run logged but no component/position info
4. Missing metrics — Run logged but no loss/accuracy
5. Stuck RUNNING runs — Run never completed (interrupted training)
6. Missing artifacts — Run finished but no report.json or checkpoint
7. Orphaned checkpoints — Checkpoint exists but no MLflow run

---

Actionable Steps

1. Check SQLite Backend

python -c "
import sqlite3
conn = sqlite3.connect('notebooks/mlruns/mlflow.db')
tables = conn.execute('SELECT name FROM sqlite_master WHERE type=\"table\"').fetchall()
print(f'Tables: {[t[0] for t in tables]}')
run_count = conn.execute('SELECT COUNT(*) FROM runs').fetchone()[0]
print(f'Run count: {run_count}')
conn.close()
"

2. List All Experiments and Runs

import mlflow
mlflow.set_tracking_uri('notebooks/mlruns')
experiments = mlflow.search_experiments()
for exp in experiments:
    runs = mlflow.search_runs(experiment_ids=[exp.experiment_id])
    print(f'{exp.name}: {len(runs)} runs')

3. Validate Run Metadata

For each run, check required fields and log completeness.

4. Cross-Reference with Checkpoints

Compare MLflow run IDs with checkpoint filenames and metadata.

5. Fix Issues

• Delete orphaned runs (with confirmation)
• Log missing parameters/metrics manually
• Rename runs to follow convention

---

Anti-Patterns to Fix (Specific)

Pattern	Where to Look	Fix
Corrupted DB	notebooks/mlruns/mlflow.db	Restore from backup or rebuild
Orphaned run	MLflow UI or search_runs	Delete or link to checkpoint
Missing params	Run metadata	Log manually via MLflow API
Stuck RUNNING	mlflow.search_runs()	Mark as FAILED
Missing artifact	Run artifact dir	Re-generate or link manually

---

Known Violations

Check these specific locations first:

• notebooks/mlruns/mlflow.db — Main MLflow database
• notebooks/mlruns/ — MLflow artifact storage
• Any run created recently may have missing metadata

---

Coverage

Already clean:

• Most runs have complete metadata
• SQLite backend is functional

Still needs work:

• Orphan detection (cross-reference with checkpoints)
• Stuck RUNNING run cleanup
• Artifact consistency checks

---

Verification

After fixing issues, verify:

# Check database health:
python -c "
import sqlite3
conn = sqlite3.connect('notebooks/mlruns/mlflow.db')
run_count = conn.execute('SELECT COUNT(*) FROM runs').fetchone()[0]
finished = conn.execute('SELECT COUNT(*) FROM runs WHERE status=\"FINISHED\"').fetchone()[0]
print(f'Runs: {run_count} total, {finished} finished')
conn.close()
"

# Check for stuck RUNNING runs:
python -c "
import mlflow
mlflow.set_tracking_uri('notebooks/mlruns')
running = mlflow.search_runs(filter_string='status=\"RUNNING\"')
print(f'Stuck RUNNING runs: {len(running)}')
"

Report Format

STATUS: [no_work | fixed]

CHANGES:
- notebooks/mlruns/mlflow.db: Cleaned up 3 orphaned runs
- Run aversion_pos2_20260624: Added missing parameters

DETAILS:
{Detailed explanation of each change}