Pick the entry point, pick the cross-validator, route the metadata,
read the report. The pipeline declaration is out of scope (see
build-ml-pipeline).
Before writing the evaluation call, output the following block
verbatim in your response. Each box must be backed by an actual
tool call or an explicit decision documented in the response.
-
skore.evaluate(...) is the entry point. It is a dispatcher
that returns the right report for the task and splitter
argument. Never hand-roll cross_val_score + manual metric
prints, and don't drop back to bare sklearn for evaluation. If you
see existing cross_val_score / cross_validate /
classification_report / mean_squared_error calls in the diff,
redirect them through skore.evaluate. Consult python-api for
the exact signature.
Always pass splitter= explicitly. When splitter= is
omitted, evaluate auto-selects: if the learner's DataOp was
declared with mark_as_X(cv=...) it reuses that cross-validator
(→ CrossValidationReport), otherwise it falls back to a single
80/20 holdout (→ EstimatorReport). This stack does not declare
cv at the X marker (build-ml-pipeline § S3), so an omitted
splitter= would silently produce a holdout instead of the
gated CV choice. Passing splitter= explicitly is what makes the
G-CV-SPLITTER decision visible, and it overrides any DataOp
cv.
Two data-passing forms — pick the one that matches the
estimator:
- sklearn-style:
skore.evaluate(estimator, X, y, splitter=...)
for any estimator whose fit is (X, y).
- env-dict-style:
skore.evaluate(learner, data={"X": X, "y": y, ...}, splitter=...) for a skrub SkrubLearner (its fit
takes a single environment dict mapping skrub.var(name=...)
names to values). This is the right form for the pipelines
produced by build-ml-pipeline.
X/y and data are mutually exclusive. The same split applies
to CrossValidationReport(...); EstimatorReport(...) uses
train_data= / test_data= for the env-dict equivalent of
X_train / y_train / X_test / y_test. The full interop
pattern (env-dict-style vs sklearn-style, how data={...} keys
map to skrub.var roots, key conventions in the Project store)
is in python-api/references/skrub_interop.md; for exact
signatures, look them up via python-api against the installed
skore version.
-
Escalate to explicit report classes only when evaluate is too
coarse. The escalation order:
EstimatorReport — single fit on a held-out set (no CV); use
when CV is wasteful (e.g., evaluating the final model on all
data after CV has already been done).CrossValidationReport — k-fold over one learner with access
to per-fold artifacts.ComparisonReport — two or more learners side-by-side.
See references/reports.md for the escalation table; defer all
API details to python-api.
-
Pick the cross-validator from the structural facts of the data
— not by default (the G-CV-SPLITTER gate). The data tells you
what splitter is correct.
The structural facts arrive at the X marker through
split_kwargs (set by build-ml-pipeline at declaration time).
Mapping rules:
split_kwargs content | Splitter |
|---|
groups | GroupKFold |
| temporal ordering | ask the user (see "Time-ordered data" below) |
| none | KFold (or RepeatedKFold for small / noisy data) |
Imbalanced classification does not change the choice — use
plain KFold / GroupKFold. See "Avoid by default" below.
Avoid by default:
- Stratified variants (
StratifiedKFold,
StratifiedGroupKFold, StratifiedShuffleSplit,
RepeatedStratifiedKFold) — they reduce across-fold variance
by construction, producing over-confident error bars on the
score. Don't reach for them on imbalance.
LeaveOneOut / LeaveOneGroupOut / LeavePGroupsOut —
high per-fold variance; aggregate hides the noise. Use
KFold / GroupKFold with 5–10 splits instead.
See references/cross-validation.md § "Avoid" for the reasoning.
Wiring details: references/metadata-routing.md.
Time-ordered data — AskUserQuestion is mandatory. When
the data is temporal, fire AskUserQuestion before picking
a splitter, with four explicit options:
TimeSeriesSplit(gap=horizon) — growing-window train,
contiguous test, embargo equal to the forecast horizon.
The safe default for any horizon-h forecasting task: it
prevents the train tail from leaking into the test head
by up to one horizon. Follow up to surface n_splits /
test_size / max_train_size.TimeSeriesSplit(gap=0) — only on the user's explicit
pick. Warn in the option description that with horizon
h > 0, the last h rows of every training fold predict
values whose target time is inside the test fold; the
reported metric is optimistic.- Custom splitter — purged-and-embargoed (finance),
blocked calendar windows, walk-forward with refit
cadence. Pick this when the time structure has more shape
than
TimeSeriesSplit captures. See
references/custom-splitter.md.
KFold ignoring time — only when the user confirms
the temporal structure shouldn't drive splitting (e.g.
the time column is a covariate but the task is treated
as IID). The skill should not recommend this option on
time-ordered data without an explicit user reason.
No silent default. Even if the data looks "obviously
TimeSeriesSplit", the user picks via AskUserQuestion.
The gap parameter is the one most often wrong by default —
TimeSeriesSplit(n_splits=5) from memory uses gap=0,
which silently leaks for any non-trivial horizon. The
structured pick exists to make that visible. Ambiguous free
text ("just pick something", "you decide") routes to a
clarifying AskUserQuestion; don't infer.
Separately, ask whether the time column should stay as a
covariate or be dropped from the feature matrix (encoders
can extract calendar patterns from a timestamp; the user's
call). This is a follow-up question, not a substitute for
the splitter pick.
If split_kwargs is empty and you cannot confirm there's
no structure (from build-time checks or from the user), do
not silently default. Return to build-ml-pipeline and ask
the user first.
-
Trust skore's metric defaults; override only on explicit user
request. skore.evaluate picks task-appropriate metrics
automatically (regression: MSE/RMSE/MAE/R²; binary: accuracy,
precision, recall, F1, ROC-AUC; multiclass: macro/micro variants;
multilabel: per-label + averages). Override only when the user
says so — e.g., "use RMSE", "report ROC-AUC". Don't pre-emptively
pin metrics or pass a scoring=... argument unless asked.
-
Custom splitter — only when sklearn doesn't have it. Examples
that justify one: purged-and-embargoed time-series CV (finance),
blocked spatial CV. The contract is small: split +
get_n_splits. See references/custom-splitter.md. Otherwise,
prefer the sklearn built-in.
