eval-templates

name	eval-templates
description	Runnable evaluation template scripts for ML tasks. Match task_type to template, adapt CONFIG, run.

Evaluation Templates

Match the plan's task_type to a template in scripts/, adapt CONFIG variables, run via run_code_in_docker.

task_type	Script	Notes
extractive_qa	eval_qa.py	Pipeline + no-answer threshold for SQuAD v2
nli, classification	eval_nli.py	Label remapping, filter label=-1
zero_shot	eval_zero_shot.py	Pipeline + candidate_labels
multilabel	eval_multilabel.py	Single-label argmax for accuracy metric
summarization	eval_summarization.py	Seq2Seq + ROUGE
ner	eval_ner.py	Token classification + seqeval, parquet fallback
vision	eval_vision.py	AutoImageProcessor, grayscale→RGB
sentiment	eval_sentiment.py	Standard classification
multiple_choice	eval_multiple_choice.py	vLLM + conditional log-likelihood

How to use

1. Read the matching template: cat scripts/eval_<type>.py
2. Adapt the CONFIG section at top (MODEL_ID, DATASET_ID, SPLIT, etc.)
3. Run via run_code_in_docker