// Evaluate a trained or downloaded language model with `lm-eval-harness` standard tasks (arc, hellaswag, gsm8k, mmlu, truthfulqa, ifeval, ...). Triggered when the user wants to benchmark, eval, or compare a model — pre- or post-training.
| name | eval-model |
| description | Evaluate a trained or downloaded language model with `lm-eval-harness` standard tasks (arc, hellaswag, gsm8k, mmlu, truthfulqa, ifeval, ...). Triggered when the user wants to benchmark, eval, or compare a model — pre- or post-training. |
Wrapper: scripts/eval_lm.py. Requires pip install 'lm-eval>=0.4.5'
(or bash bootstrap.sh --eval).
python scripts/eval_lm.py \
--model HuggingFaceTB/SmolLM2-360M-Instruct \
--tasks arc_easy,hellaswag,piqa \
--num-fewshot 0 \
--batch-size auto
For local checkpoints, pass the directory (outputs/sft-default) or any Hub
repo id. lm-eval handles loading.
| Goal | Tasks |
|---|---|
| Instruct-quality smoke | arc_easy,hellaswag,piqa,winogrande |
| Math reasoning | gsm8k,minerva_math |
| Knowledge | mmlu |
| Truthfulness | truthfulqa_mc2 |
| Instruction following | ifeval |
| Code | humaneval (needs sandbox) |
After the run, surface:
research-recipe output should include
these).Do not just dump the JSON. Pick the 3-5 numbers the user actually cares about and put them in a small table.
Set up training and inference on AMD Ryzen AI Max+ 395 / Strix Halo (gfx1151, RDNA 3.5) with TheRock nightly ROCm wheels. Triggered when the host has gfx1151, when `rocminfo` shows Strix Halo, or when the user mentions Strix Halo / Ryzen AI Max / gfx1151 / 128GB unified memory.
Recreate the ml-intern-kit Python environment on a new machine (laptop, rented GPU box, Docker, fresh checkout). Triggered when the user is on a new host, sees ImportError on a core dep (torch/transformers/trl/peft/accelerate), or wants to install flash-attn / unsloth / bitsandbytes after the fact.
Audit a Hugging Face dataset before training to confirm splits, columns, format, sample rows, distributions, and duplicates. Triggered before any training/fine-tuning script runs, when a user mentions a new dataset, or when you hit a KeyError / format mismatch in a training job.
Submit a training/inference script to Hugging Face Jobs (`hf jobs run`). Triggered when the user wants to run training in the cloud, scale beyond local hardware, or kick off a multi-hour fine-tune. Enforces pre-flight: hub_model_id required, ≥2h timeout for training, single-job validation before batch.
Run a literature-first crawl before writing ANY ML training/fine-tuning/inference code. Spawns an Explore sub-agent that mines papers, citation graphs, methodology sections, and matched HF datasets to produce a ranked list of training recipes attributed to specific published results. Triggered when the user asks to fine-tune, train, or improve a model, or when the user names a task/benchmark and you need a recipe before coding.
Direct Preference Optimization (DPO) fine-tune with TRL `DPOTrainer`. Triggered when the user wants to align a model on preferences / pairwise comparisons / chosen-vs-rejected data, or improve an existing SFT checkpoint with a preference dataset.
