| name | ai-ml-build-ml-data-pipelines |
| description | Use when building datasets, labels, features, embeddings, fine-tuning data, training or inference pipelines, experiment tracking, model artifacts, data validation, or reproducible ML workflows; applies the Microsoft ML lifecycle checklist on top of core data engineering. |
ML Data Pipelines
Apply core state/data, security, contracts, testing, and release rules. Read
references/source-notes-microsoft-ml-playbook.md
for the lifecycle and production checklist.
Gate investment with evidence
Envision the business problem and intended use, then run a time-boxed feasibility study. Confirm
data access, rights, lineage, quality, representativeness, labels or ground truth, baseline, minimum
useful performance, integration path, responsible-AI risks, and ROI. End with proceed, pivot, or
stop plus evidence; experimentation is not an automatic commitment to production.
Version the complete experiment
For every run, bind code revision, environment/dependencies, source dataset and lineage, split
method, preprocessing/features, prompt or training configuration, hyperparameters, random seeds,
model/provider version, evaluation code/version, metrics, and artifacts. Keep immutable raw inputs
where policy allows and derive processed data through reproducible pipelines. Prevent entity,
temporal, duplicate, and target leakage across splits.
Separate exploration from production while sharing tested transformation and evaluation modules.
Notebooks may explore; scheduled and serving pipelines use reviewable modules with stable inputs,
outputs, exceptions, and tests.
Build trustworthy pipelines
- Validate schema, ranges, missingness, distributions, freshness, duplicates, label quality, tenant,
consent, and privacy before training, indexing, or scoring.
- Make transformations deterministic where possible and safe to rerun. Track partial progress and
quarantine invalid data instead of silently coercing it.
- Version embeddings with source content, chunking, preprocessing, model, dimensions, and index
configuration. Define refresh, deletion propagation, tenant isolation, and full rebuild.
- Fine-tune only when evals show prompting/retrieval cannot meet a stable repeated behavior and the
team has representative licensed data, a baseline, safety review, and maintenance capacity.
- Register deployable artifacts with lineage, evaluation, approval, compatibility, and rollback.
Operationalization gate
Require a meaningful baseline, consistent evaluation, tested stable schemas, end-to-end pipeline
proof, hard quality/latency/cost limits, input and drift monitoring, obtainable production ground
truth or proxy, responsible-AI review, recovery, accountable owner, and retraining/replacement
policy. Use ai-ml-operate-model-monitoring after deployment.
