validate-datasets

name	validate-datasets
description	Dataset management maintenance — validates dataset integrity, version tracking, storage health, and cache consistency for training datasets.

Validate Datasets

Mental Model

Datasets are the training data foundation. When datasets are corrupted, versioned incorrectly, or have integrity issues, training produces garbage results. This skill ensures every dataset is loadable, properly versioned, and consistent with the training notebooks that use them.

Coverage

Documented: Dataset integrity, version tracking, storage health, cache consistency, notebook compatibility. Not yet documented: Dataset drift detection, label quality checks, train/val split validation. Last extended: 2026-06-24

---

What This Skill Checks

1. Dataset Integrity

Every dataset in resources/datasets/ must be loadable.

Verification command:

python -c "
import datasets
# Test BeaverTails
ds = datasets.load_from_disk('resources/datasets/beavertails')
print(f'BeaverTails: {len(ds)} samples')
# Test SocialIQa
ds = datasets.load_from_disk('resources/datasets/social_iqa')
print(f'SocialIQa: {len(ds)} samples')
print('Datasets OK')
"

Checks:

• Dataset files exist on disk
• Dataset can be loaded with datasets.load_from_disk()
• Dataset has expected number of samples
• Dataset has expected columns/features

2. Dataset Specifications

Each dataset used in training should have a spec in notebooks/components/data_loading.py:

Dataset	Expected Samples	Expected Columns	Purpose
BeaverTails	~30K (30K split)	prompt, response, label	Aversion training
SocialIQa	~74K	context, question, answerA/B/C, label	Empathy training

Checks:

• Spec exists in data_loading.py
• Spec matches actual dataset on disk
• Spec matches notebook usage

3. Version Tracking

Datasets should have version information:

Checks:

• Dataset source is documented (HuggingFace dataset name)
• Dataset split is documented (train, validation, test)
• Dataset size is documented (number of samples)
• Any preprocessing is documented

4. Storage Health

Checks:

• Dataset files are not empty
• Dataset files are reasonable size (BeaverTails ~100MB, SocialIQa ~200MB)
• No orphaned dataset files (not used by any notebook)
• No missing dataset files (used by notebook but not on disk)

5. Cache Consistency

HuggingFace datasets can be cached in multiple locations:

Checks:

• Primary dataset in resources/datasets/
• No conflicting cached versions in ~/.cache/huggingface/
• Dataset hash matches expected value (if documented)

6. Notebook Compatibility

Each notebook that uses a dataset should be compatible:

Checks:

• Notebook imports data_loading module
• Notebook uses the correct dataset spec
• Notebook handles missing dataset gracefully
• Notebook doesn't hardcode dataset paths

---

Anti-Patterns to Fix

1. Corrupted dataset — File exists but can't be loaded
2. Missing dataset — Notebook requires dataset but it's not on disk
3. Wrong version — Dataset loaded but has wrong columns/features
4. Orphaned dataset — Dataset exists but no notebook uses it
5. Hardcoded paths — Notebook uses "/path/to/dataset" instead of data_loading module
6. Missing spec — Dataset used but no spec in data_loading.py
7. Cache conflict — Multiple cached versions of same dataset

---

Actionable Steps

1. List All Datasets

ls -la resources/datasets/

2. Test Dataset Loading

import datasets, os
for name in os.listdir('resources/datasets'):
    path = f'resources/datasets/{name}'
    if os.path.isdir(path):
        try:
            ds = datasets.load_from_disk(path)
            print(f'OK: {name} ({len(ds)} samples)')
        except Exception as e:
            print(f'FAIL: {name}: {e}')

3. Check Dataset Specs

Read notebooks/components/data_loading.py and verify all specs are documented.

4. Cross-Reference with Notebooks

Check which notebooks use which datasets and verify compatibility.

5. Fix Issues

• Re-download corrupted datasets
• Add missing specs to data_loading.py
• Remove orphaned datasets (with confirmation)
• Update hardcoded paths to use data_loading module

---

Anti-Patterns to Fix (Specific)

Pattern	Where to Look	Fix
Corrupted dataset	resources/datasets/	Re-download from source
Missing dataset	Notebook imports	Download or create spec
Hardcoded path	Notebook code	Use data_loading module
Missing spec	data_loading.py	Add dataset specification
Orphaned dataset	resources/datasets/	Delete or document usage
Cache conflict	~/.cache/huggingface/	Clear conflicting cache

---

Known Violations

Check these specific locations first:

• resources/datasets/ — All cached datasets
• notebooks/components/data_loading.py — Dataset specifications
• All training notebooks — Dataset usage patterns

---

Coverage

Already clean:

• Most datasets are loadable
• Most notebooks use data_loading module

Still needs work:

• Version tracking documentation
• Orphan detection
• Cache consistency checks

---

Verification

After fixing issues, verify:

# Check all datasets load:
python -c "
import datasets, os
for name in os.listdir('resources/datasets'):
    path = f'resources/datasets/{name}'
    if os.path.isdir(path):
        ds = datasets.load_from_disk(path)
        print(f'OK: {name} ({len(ds)} samples)')
"

# Check no hardcoded paths:
grep -r "resources/datasets/" notebooks/*.py | grep -v "data_loading"

# Check specs exist:
grep -c "def.*dataset" notebooks/components/data_loading.py

Report Format

STATUS: [no_work | fixed]

CHANGES:
- resources/datasets/beavertails: Re-downloaded corrupted dataset
- notebooks/components/data_loading.py: Added spec for social_iqa

DETAILS:
{Detailed explanation of each change}