| name | eval-model |
| description | Evaluate a trained or downloaded language model with `lm-eval-harness` standard tasks (arc, hellaswag, gsm8k, mmlu, truthfulqa, ifeval, ...). Triggered when the user wants to benchmark, eval, or compare a model — pre- or post-training. |
eval-model — lm-eval-harness wrapper
Wrapper: scripts/eval_lm.py. Requires pip install 'lm-eval>=0.4.5'
(or bash bootstrap.sh --eval).
Quick eval
python scripts/eval_lm.py \
--model HuggingFaceTB/SmolLM2-360M-Instruct \
--tasks arc_easy,hellaswag,piqa \
--num-fewshot 0 \
--batch-size auto
For local checkpoints, pass the directory (outputs/sft-default) or any Hub
repo id. lm-eval handles loading.
Task picks by training goal
| Goal | Tasks |
|---|
| Instruct-quality smoke | arc_easy,hellaswag,piqa,winogrande |
| Math reasoning | gsm8k,minerva_math |
| Knowledge | mmlu |
| Truthfulness | truthfulqa_mc2 |
| Instruction following | ifeval |
| Code | humaneval (needs sandbox) |
Reporting
After the run, surface:
- Task → score → baseline comparison. Use the published numbers from the
paper that motivated the recipe (the
research-recipe output should include
these).
- The model URL on the Hub.
- The Trackio / wandb run URL for training, if applicable.
Do not just dump the JSON. Pick the 3-5 numbers the user actually cares about
and put them in a small table.
