| name | ml-research |
| description | Empirical machine-learning claims that survive scrutiny: a baseline strong enough to be worth beating, splits that respect grouping and time, leakage hunted before any number is trusted, ablations that isolate one change, results reported as a spread over seeds rather than a single number, and benchmark deltas tested instead of eyeballed. Use when the task is measuring whether a model or a change actually works.
|
| version | 1.0.0 |
| author | Remedy |
| tags | ["machine-learning","research","evaluation","benchmarking","reproducibility"] |
| requires | [] |
| tools | ["data_profile","data_diff","analysis_run","analysis_ledger","analysis_env","stats_effect_size","stats_multiplicity","lit_search","cite_add","skill_activate"] |
| triggers | ["\\b(train(?:ing)?[/ -]?(?:test|val|validation)[/ -]?split|cross[- ]?validat\\w+|hold[- ]?out set|data leakage)\\b","\\b(hyper-?parameter (?:search|tuning|sweep)|ablation (?:study|table|experiment)|learning curves?)\\b","\\b(ROC[- ]?AUC|\\bAUROC\\b|F1 scores?|precision[/ ]recall|calibration (?:curve|error)|\\bECE\\b)\\b","\\b(random seeds?|seed variance|overfit\\w*|generali[sz]ation gap|model card|benchmark leaderboard)\\b"] |
ML research
skill_activate(skill="research-method") first, and do not restate the
spine — that pack owns question framing, evidence standards,
preregistration, citation honesty and "we do not know". This pack owns
what is different about research on models: the claim is empirical, the
measurement is noisy, and almost every published failure is a
measurement failure, not a modelling one.
Shipping a model is a different job. Here the deliverable is a claim
("X improves Y by Z under conditions C") and the evidence that supports
it.
Order of work
- Write the claim and its falsifier first. "Method A beats B on
task T by more than the seed spread, under an equal tuning budget."
If you cannot say what result would refute it, the experiment is not
designed yet.
analysis_env(path) — what runs here, which frameworks and
versions, is there a GPU, is pandas/numpy importable in the project
env. Everything after this goes through analysis_run so the argv,
input hashes, artifacts and duration land in the ledger.data_profile(path, target=<label>) before any training. Read
leakage_suspects and the class balance. Duplicated rows across a
split are the most common way a paper's headline number is wrong.- Split before you look —
references/splits-and-leakage.md. Group
by the unit that must not straddle the boundary (patient, user,
document, session, molecule scaffold); split by time when deployment
is forward in time. Freeze the test set and touch it once.
- Build the honest baseline —
references/baselines-and-ablations.md.
Majority class, a linear/gradient-boosted model on the same features,
and the prior work's method tuned as hard as yours. A baseline you
did not tune is not a baseline.
- Run with N seeds, not one —
references/variance-and-seeds.md.
Report mean and spread (SD or min-max over seeds), never a single
number, and never the best seed.
- Ablate one thing at a time from the full system, each with the
same seeds and budget.
- Evaluate deliberately —
references/evaluation-design.md: the
metric that matches the decision, calibration, subgroup breakdowns,
and a real test of the delta (stats_effect_size, paired across seeds
or bootstrap over the test set; stats_multiplicity when many
variants are compared).
- Report —
references/benchmark-reporting.md and
.
