| name | kaggle-cv-setup |
| description | Create or validate the shared cross-validation split (artifacts/folds.parquet) that every model must use, and check that it tracks the leaderboard. Use before any model trains, or whenever CV and LB diverge. |
Skill: shared CV setup & validation
The single most important artifact in the kit. Every model trains on this exact split so their
OOF predictions align and can be blended.
Create the split
python src/make_folds.py
Writes artifacts/folds.parquet = {id, fold} using fold_scheme from config.yaml:
stratified — default for multiclass (class-balanced folds; independent rows).stratified_group — class-balanced AND keeps each group_col entity in one fold (no entity leak).group — grouped, balance not critical.time — forward-chaining folds for temporal data.
Validate the split
- Class balance per fold — printed by
make_folds.py; each fold should mirror the global
class distribution (except intentionally for group/time).
- No entity leak — for grouped data, confirm no
group_col value appears in two folds.
- CV↔LB gap — after the first real submission, compare CV to LB. A stable gap (they move
together) means CV is trustworthy. A shifting/inverted gap means the split is wrong.
If CV doesn't track LB — fix the split first
Before tuning anything else, diagnose:
- Is there a hidden group (same entity in many rows) you're splitting across folds? →
stratified_group.
- Is the test set later in time? →
time folds.
- Is there train/test drift (adversarial AUC ≫ 0.5)? → the CV distribution differs from test;
consider adversarial validation weighting or dropping unstable features.
- Is a leak inflating CV? → audit target-based features for out-of-fold correctness.
Guardrails
- One canonical split for cross-model blending. You may repeat with a second fold-seed to
shrink final-model variance, but pick ONE split as the alignment reference for OOF.
- Never let a model re-fold on its own. If
folds.parquet changes, all OOF must be regenerated.
