| name | sdv-build-data |
| description | Use when building or extending a dataset in an SDV `-data` repo (nfl-data, cfb-data, hoopR-*-data, wehoop-*-data, fastRhockey-*-data) — the ingest → build → validate → publish pipeline that reads the sibling `-raw` repo tree, writes tidy parquet, and publishes via git commit + per-file gh release upload. Invoke for "build the dataset", "add a data-repo builder", "publish to the release", "reshape raw to data", or "daily processor". |
Build-data — the -raw → -data → release pipeline
SDV splits producer repos: -raw repos scrape and commit raw JSON
(scraping-only — never reintroduce ML/model deps there); -data repos own
reshaping, modeling, reports, and publishing. This skill encodes the data
side: read the sibling raw tree, build tidy frames, validate, publish.
Repo map (data repo ← raw source ← publish target)
| Data repo | Reads from | Package layout | Publishes to |
|---|
nflverse-dev/nfl-data | nfl-raw nfl/raw/{season}/{game_id}.json | python/{nfl_data_ingest, native_pbp, nfl_model_publish, model_training} | sportsdataverse-data releases (nfl_model_pbp, nfl_model_artifacts, nfl_4th_down_models) |
cfbfastR-dev/cfbfastR-cfb-data | cfbfastR-cfb-raw cfb/{json,…} | python/{cfb_data_ingest, cfb_data_build, cfb_model_pbp, cfb_model_publish, cfb_model_reports} + R producer | espn_cfb_* releases (14 public datasets; R is the released producer — python builders must parity-match) |
hoopR-dev/hoopR-nba-stats-data | hoopR-nba-stats-raw | python/nba_data_build | git commit + sportsdataverse-data release tags (e.g. nba_stats_pbpv3, possessions_v3, lineups_v3) |
hoopR-dev/hoopR-{mbb,nba}-data, wehoop-dev/wehoop-*-data, hockey-dev/fastRhockey-*-data | family -raw mirror | family-shaped | family releases |
Cron entry point: scripts/daily_<family>_processor.sh (data side) mirrors
the raw side's daily_<family>_scraper.sh. The target repo's CLAUDE.md +
existing package layout govern over this table — read them first.
Pipeline steps
- Ingest — read the sibling
-raw checkout directly from disk (it's a
sibling under GitHub-Data/; never re-scrape or clone). Path via the
repo's config/env (e.g. SDV_VALIDATION_*_DATA_ROOT patterns), not
hardcoded absolute paths. Put reading logic in <x>_data_ingest/.
- Build — a builder module per dataset in
<x>_data_build/ (mirror the
existing builders' signature/CLI). polars 1.x; snake_case columns; one
canonical dtype per id at the boundary; empty frames carry the documented
schema; partition output like the released dataset (per-season parquet is
the norm).
- Validate before publish — run the validation-harness checks the repo
wires (constant-column, extraction coverage, parity vs the PRIOR release
for overlapping seasons). Where an R producer released the dataset
(cfb-data), the python build must parity-match the R-released parquet
before it ships. The harness has caught real producer bugs — do not skip.
- Publish — two mirrors, both required when the repo uses both:
- git commit of the data tree (batch, conventional message);
- release assets:
gh release upload <tag> <one-file> --clobber —
per-file loop, never a multi-file glob (multi-asset uploads
silently drop large files). Respect the repo's release map — e.g.
nfl nfl_model_publish has DECISION_MODELS_RELEASE_MAP (fd →
nfl_4th_down_models, rest → nfl_model_artifacts) that the generic
subcommand ignores; check where each artifact belongs before uploading.
- Wire the cron — add/extend
scripts/daily_<family>_processor.sh so
the daily run picks the new dataset up; log to logs/ with the
scrape-job conventions (unbuffered, timestamped, EXIT=$?).
Gotchas
- Raw/data boundary is one-way: data repos read raw trees; raw repos
never import from data repos and never grow modeling deps.
- Loader handoff: if sdv-py (or an R package) should load the new
dataset, that's a separate consumer-side PR (cached loader + returns
schema) — note it as follow-up, don't bundle it into the producer change.
- New master-flag columns: upserting a new boolean flag into a
*_schedule_master.parquet whose schema predates it crashes on first
run — add the column with a default before the join.
- Piggyback releases: several families publish onto an EXISTING release
tag (
espn_cfb_*, sportsdataverse-data) rather than cutting new ones —
find the tag the released dataset already lives under.
