check-dependencies

name	check-dependencies
description	Comprehensive dependency maintenance — verifies import health, requirements.txt consistency, import hierarchy, and environment health.

Check Dependencies

Mental Model

Dependencies are the project's external contracts. Every import statement is a trust decision — you're pulling in code that runs in your research environment. This skill ensures the dependency graph is consistent, importable, and healthy.

Coverage

Documented: Import health, requirements.txt consistency, import hierarchy, environment health. Not yet documented: Automated dependency updates, vulnerability scanning. Last extended: 2026-06-24

---

What This Skill Checks

1. Import Health

Every Python file in the project should import successfully. The actual file locations are:

Standard notebooks (config + single run_standard_experiment() call):

• notebooks/01_train_aversion_layer.py
• notebooks/02_train_empathy_layer.py
• notebooks/03_train_coherence_layer.py
• notebooks/04_train_humility_layer.py

Experimental notebooks (custom training loops — import torch, datasets, sklearn, etc.):

• notebooks/01b_teacher_distillation.py
• notebooks/01c_aversion_sweep.py
• notebooks/01d_centroid_steering.py

Shared components (the ML infrastructure layer):

• notebooks/components/*.py — 10 modules (experiment_runner, training_loop, training_rig, data_loading, artifacts, device_setup, mlflow_tracking, html_report, run_comparison, init)

Technique modules (fully self-contained, no cross-imports):

• resources/techniques/*.py — ~50 modules

Utilities:

• notebooks/utils/vram_monitor.py

Root scripts:

• run.py

Note: Files that assert device.type == "cuda" (like device_setup.py) will fail on CPU-only machines. These are expected failures — report them but don't treat them as bugs. Standard notebooks that call device_setup.setup() will also fail without GPU; the import check should verify they parse correctly, not that they run.

2. Requirements Consistency

The project uses a flat requirements.txt (not pip-compile, not Poetry). Current contents:

# Core ML
torch>=2.6.0
torchvision>=0.21.0
transformers==5.5.3
datasets==4.8.4

# Analysis & visualization
scikit-learn>=1.8.0
matplotlib>=3.10.0
numpy>=2.4.0
tqdm>=4.67.0

# Experiment tracking
mlflow
pyyaml

# Music-similarity pipeline (MS series)
onnxruntime-gpu>=1.20.0
duckdb>=1.1.0
librosa>=0.10.0
requests>=2.32.0

Known issues to check:

• mlflow and pyyaml have no version pins — these should be pinned for reproducibility
• torch and torchvision use >= pins — acceptable for CUDA-dependent packages but should be tested against upper bounds
• transformers and datasets use exact pins (==) — correct for research reproducibility
• The music-similarity packages (onnxruntime-gpu, duckdb, librosa) may not be needed by the core PDLI notebooks — check if they're actually imported by the standard notebooks

Import-to-requirements mapping (what the project actually imports):

Import name	requirements.txt name	Notes
torch	torch	Core
torchvision	torchvision	Core
transformers	transformers	Core
datasets	datasets	Core
sklearn	scikit-learn	Name mismatch (pip name vs import name)
matplotlib	matplotlib	Core
numpy	numpy	Core
tqdm	tqdm	Core
mlflow	mlflow	Tracking
yaml	pyyaml	Name mismatch (pip name vs import name)
onnxruntime	onnxruntime-gpu	Name mismatch (pip name vs import name)
duckdb	duckdb	Music pipeline
librosa	librosa	Music pipeline
requests	requests	HTTP

Note on name mismatches: sklearn imports from scikit-learn, yaml imports from pyyaml, onnxruntime imports from onnxruntime-gpu. These are standard Python packaging conventions, not bugs.

3. Cross-Layer Import Rules

Enforce the project's import hierarchy:

• notebooks/components/ must NOT import from notebooks/*.py (circular risk — components are the dependency, notebooks are the consumers)
• resources/techniques/ must NOT import from other technique modules (self-containment rule — each technique is standalone)
• notebooks/ should import from components/, not vice versa
• notebooks/components/ CAN import from notebooks/utils/ (utilities are shared infrastructure)
• resources/techniques/ CAN import from torch, numpy, and other third-party packages

Actual import patterns observed:

• Standard notebooks: from components import data_loading and from components.experiment_runner import run_standard_experiment
• Experimental notebooks: Direct imports from torch, transformers, datasets, sklearn, matplotlib
• Components: Import from torch, pathlib, json, and each other within components/

4. Type Checking

The project has two type checkers configured, both lenient:

mypy (mypy.ini):

[mypy]
python_version = 3.12

[mypy-datasets.*]
ignore_missing_imports = True

pyright (pyrightconfig.json):

{
  "reportMissingModuleSource": false,
  "reportMissingTypeStubs": false,
  "reportMissingImports": false
}

Both are configured to be lenient — missing imports are suppressed because many ML packages lack type stubs. This is intentional for an ML research project.

Run commands:

python -m mypy --config-file mypy.ini notebooks/components/
python -m pyright --project pyrightconfig.json

Report findings but don't auto-fix type errors — they often require design decisions.

---

Anti-Patterns to Fix

Pattern	Where to Look	Fix
Missing package	requirements.txt	Add with version pin
Unused package	requirements.txt	Remove (confirm first)
Cross-layer import	notebooks/*.py importing from notebooks/*.py	Refactor to use components/
Import failure	Various	Investigate — may be expected (GPU requirement)
Version conflict	requirements.txt	Resolve compatibility
Unpinned version	requirements.txt lines without == or >=	Pin to specific version or range
Circular import	notebooks/components/*.py	Check with static analysis

---

Actionable Steps

1. Scan for Python files

Use glob to find all .py files, excluding .venv/ and code-intel/:

# Core files to check:
notebooks/*.py
notebooks/components/*.py
notebooks/utils/*.py
resources/techniques/*.py
run.py

2. Parse imports from each file

Extract import X and from X import Y statements. Filter to third-party packages (skip stdlib modules like os, sys, json, pathlib, time, random, datetime, glob, subprocess, argparse).

3. Compare with requirements.txt

For each third-party import, check if the corresponding package is in requirements.txt. Remember the name mismatches:

• sklearn → scikit-learn
• yaml → pyyaml
• onnxruntime → onnxruntime-gpu

4. Check import hierarchy

Verify no circular or cross-layer imports:

• notebooks/components/*.py should NOT import from notebooks/*.py
• resources/techniques/*.py should NOT import from other resources/techniques/*.py

5. Run type checks

python -m mypy --config-file mypy.ini notebooks/components/
python -m pyright --project pyrightconfig.json

6. Fix issues

• Add missing packages to requirements.txt with version pins
• Remove unused packages (confirm with user first)
• Pin unpinned versions (especially mlflow and pyyaml)
• Report issues requiring manual intervention

---

Verification

After fixing issues, verify the changes:

# Verify requirements.txt installs cleanly:
pip install -r requirements.txt

# Verify no broken imports in the project:
python -c "
import importlib, pathlib, sys
for p in pathlib.Path('notebooks/components').glob('*.py'):
    mod = f'notebooks.components.{p.stem}'
    try:
        importlib.import_module(mod)
        print(f'OK: {mod}')
    except Exception as e:
        print(f'FAIL: {mod} — {e}')
"

# Verify technique modules are self-contained:
for f in resources/techniques/*.py; do
  python -c "import ast; tree = ast.parse(open('$f').read()); imports = [n for n in ast.walk(tree) if isinstance(n, (ast.Import, ast.ImportFrom))]; [print(f'  {n.module}') for n in imports if isinstance(n, ast.ImportFrom) and n.module and 'resources.techniques' in n.module]" 2>/dev/null
done

---

Report Format

STATUS: [no_work | fixed]

CHANGES:
- requirements.txt: Added `requests>=2.31.0`
- requirements.txt: Removed `unused-package`
- requirements.txt: Pinned `mlflow==2.15.0`

ISSUES (require manual fix):
- notebooks/01b_teacher_distillation.py: imports `something` not in requirements.txt
- notebooks/components/artifacts.py: potential circular import with training_rig

DETAILS:
{Detailed explanation of each change}