model-training

name	model_training
description	Use for tasks that train, fine-tune, adapt, or evaluate predictive models: AutoML tabular models, time-series forecasting, custom PyTorch modules, object detection, QA fine-tuning, LoRA/support models, checkpoints, predictions, and metrics files.

Model Training Dispatch

Model-training tasks come in shapes that share almost nothing in common. The first job is to identify the shape, because every later choice — library, splits, artifacts, validation — flows from it. This guide is a dispatch table: locate the shape, follow the section.

Identify the shape

Read instruction.md and the data tree, then answer:

Signal	Shape
Tabular CSV/Parquet, fixed feature columns, target column named	Tabular AutoML
Date/time index per entity, requested horizon, frequency	Forecasting
Custom architecture, train loop, GPU mention	Custom NN
Pretrained backbone + task-specific head, prompt/response pairs	Fine-tuning
Adapter weights only, frozen base model	LoRA / parameter-efficient
Reward signal, policy update	RL adaptation

Cross-cutting invariants for every shape:

• Lock data splits exactly as instructed. Splits are often positional or chronological — do not shuffle unless told to.
• Set seeds for random, numpy, and any model framework. Determinism is the default; deviate only when explicitly required.
• Save the artifacts the task names, at the paths it names, with the formats it names. Conveniently named alternatives fail tests.
• Prefer CPU-safe settings. GPU is opt-in per task.

Tabular AutoML

LightAutoML for binary/multiclass/regression on tabular data. The library returns a NumpyDataset from predict(); remember to slice probabilities.

import json
import pandas as pd
from lightautoml.automl.presets.tabular_presets import TabularAutoML
from lightautoml.tasks import Task
from sklearn.metrics import roc_auc_score

df = pd.read_csv("<dataset>.csv")
train, test = df.iloc[:<train_n>].copy(), df.iloc[<train_n>:].copy()
task = Task("binary")  # "reg" or "multiclass" also supported
roles = {"target": "<target_column>"}

automl = TabularAutoML(task=task, timeout=120, cpu_limit=2)
automl.fit_predict(train, roles=roles)
pred = automl.predict(test).data[:, 0]

pd.DataFrame({"index": test.index, "prediction": pred}).to_csv(
    "predictions.csv", index=False
)
with open("metrics.json", "w") as f:
    json.dump(
        {"roc_auc": roc_auc_score(test["<target_column>"], pred), "n_test": len(test)},
        f,
    )

Notes:

• predict() returns 2D NumpyDataset.data; .data[:, 0] for binary positive-class probabilities.
• Without timeout, training is unbounded. Always set it.
• cpu_limit matters when the runner reserves cores; match the task instruction.
• Keep verbose=0 unless debugging.

Forecasting

Multi-series demand or load forecasting (tsururu, sktime, statsforecast).

• Preserve entity IDs, horizon, frequency, lag features, and chronological splits.
• Never shuffle. Train cutoff < test start, always.
• Evaluate with the requested metric (SMAPE, MAE, MAPE, MASE) — direction and rounding matter for the grade.
• Output one prediction file with (entity_id, timestamp, forecast) columns unless instructed otherwise.

Custom Neural Networks

For PyTorch tasks where the architecture is part of the deliverable:

• Set seeds for random, numpy, torch, torch.cuda (if used).
• Keep the model class importable from the expected module path; tests load it.
• Save state dicts at the requested file path. Full pickled models are rarely the right deliverable.
• Guard the training loop against NaN loss. Stop and report rather than save a NaN checkpoint.
• Evaluate inside model.eval() and torch.no_grad().

Fine-tuning (QA, classification, generation)

Adapter or full-tune of a pretrained backbone.

• Match prompt/response formatting exactly — extra newlines or wrong delimiters silently degrade scores.
• Tokenizer truncation rules: keep the answer span intact for QA; truncate from the front for long contexts when allowed.
• Train/validation split as instructed; many tasks pin specific row ranges.
• Save tokenizer and config alongside weights, or the model is unusable from disk.

LoRA / parameter-efficient

• Save adapter weights plus the LoRA config, not the merged base model — the task tests reapply the adapter to the original base.
• Record which layers were adapted (target_modules, rank, alpha). Tests sometimes assert these.
• Confirm the adapter loads cleanly on the original base before declaring done.

RL adaptation

• Record reward curves and the seed; reproducibility is judged on these.
• Save final policy weights at the requested path.
• Evaluate against the same fixed prompt set or environment seed used at training.

Validation

For every shape:

• Re-read predictions and metrics from disk; check schema, row count, numeric range.
• Confirm split sizes; rule out target leakage by checking train/test column distributions.
• Round-trip the saved model: load from disk, run a small inference batch, compare to in-memory prediction.
• Compare artifact filenames to the task spec, not to convenient defaults.

Pitfalls

• Shuffling when split is positional or chronological — common cause of "great score, fails tests".
• Saving class labels where probabilities are required (or vice versa).
• Dropping original index in prediction files — joins downstream fail.
• Producing a model artifact that cannot be reloaded without local state (in-memory tokenizers, custom classes not importable).
• Letting expensive training exceed the benchmark timeout — kill-by-timeout looks like a code crash.
• Saving a merged-base+LoRA file when the task wanted adapter-only.