// Runnable evaluation template scripts for ML tasks. Match task_type to template, adapt CONFIG, run.
Create and manage datasets on Hugging Face Hub. Supports initializing repos, defining configs/system prompts, streaming row updates, and SQL-based dataset querying/transformation. Designed to work alongside HF MCP server for comprehensive dataset workflows.
Add and manage evaluation results in Hugging Face model cards. Supports extracting eval tables from README content, importing scores from Artificial Analysis API, and running custom model evaluations with vLLM/lighteval. Works with the model-index metadata format.
| name | eval-templates |
| description | Runnable evaluation template scripts for ML tasks. Match task_type to template, adapt CONFIG, run. |
Match the plan's task_type to a template in scripts/, adapt CONFIG variables, run via run_code_in_docker.
| task_type | Script | Notes |
|---|---|---|
| extractive_qa | eval_qa.py | Pipeline + no-answer threshold for SQuAD v2 |
| nli, classification | eval_nli.py | Label remapping, filter label=-1 |
| zero_shot | eval_zero_shot.py | Pipeline + candidate_labels |
| multilabel | eval_multilabel.py | Single-label argmax for accuracy metric |
| summarization | eval_summarization.py | Seq2Seq + ROUGE |
| ner | eval_ner.py | Token classification + seqeval, parquet fallback |
| vision | eval_vision.py | AutoImageProcessor, grayscale→RGB |
| sentiment | eval_sentiment.py | Standard classification |
| multiple_choice | eval_multiple_choice.py | vLLM + conditional log-likelihood |
cat scripts/eval_<type>.py