eval-templates

Étoiles38

Forks3

Mis à jour12 mai 2026 à 23:06

Runnable evaluation template scripts for ML tasks. Match task_type to template, adapt CONFIG, run.

Installation

Installer avec Codex ou Claude Copiez ce prompt, collez-le dans Codex, Claude ou un autre assistant, puis laissez-le vérifier la page du skill et l'installer pour vous.

Exécuter dans Manus

Source

allenai

allenai/artifact-linker

Ouvrir le dépôt GitHub Voir les dépôts du créateur

Téléchargement

Exécuter dans Manus

Métiers associésSOC

Basé sur la classification professionnelle SOC

Scientifiques des donnéesProfessions informatiques et mathématiques·SOC 15-2051

Explorateur de fichiers

10 fichiers

SKILL.md

readonly

name	eval-templates
description	Runnable evaluation template scripts for ML tasks. Match task_type to template, adapt CONFIG, run.

Evaluation Templates

Match the plan's task_type to a template in scripts/, adapt CONFIG variables, run via run_code_in_docker.

task_type	Script	Notes
extractive_qa	eval_qa.py	Pipeline + no-answer threshold for SQuAD v2
nli, classification	eval_nli.py	Label remapping, filter label=-1
zero_shot	eval_zero_shot.py	Pipeline + candidate_labels
multilabel	eval_multilabel.py	Single-label argmax for accuracy metric
summarization	eval_summarization.py	Seq2Seq + ROUGE
ner	eval_ner.py	Token classification + seqeval, parquet fallback
vision	eval_vision.py	AutoImageProcessor, grayscale→RGB
sentiment	eval_sentiment.py	Standard classification
multiple_choice	eval_multiple_choice.py	vLLM + conditional log-likelihood

How to use

Read the matching template: cat scripts/eval_<type>.py
Adapt the CONFIG section at top (MODEL_ID, DATASET_ID, SPLIT, etc.)
Run via run_code_in_docker

Plus depuis ce dépôt

même dépôt

hugging-face-dataset-viewer

allenai/artifact-linker

Use this skill for Hugging Face Dataset Viewer API workflows that fetch subset/split metadata, paginate rows, search text, apply filters, download parquet URLs, and read size or statistics.

2026-05-1238

hugging-face-datasets

allenai/artifact-linker

Create and manage datasets on Hugging Face Hub. Supports initializing repos, defining configs/system prompts, streaming row updates, and SQL-based dataset querying/transformation. Designed to work alongside HF MCP server for comprehensive dataset workflows.

2026-05-1238

hugging-face-evaluation

allenai/artifact-linker

Add and manage evaluation results in Hugging Face model cards. Supports extracting eval tables from README content, importing scores from Artificial Analysis API, and running custom model evaluations with vLLM/lighteval. Works with the model-index metadata format.

2026-05-1238

name	eval-templates
description	Runnable evaluation template scripts for ML tasks. Match task_type to template, adapt CONFIG, run.

Evaluation Templates

Match the plan's task_type to a template in scripts/, adapt CONFIG variables, run via run_code_in_docker.

task_type	Script	Notes
extractive_qa	eval_qa.py	Pipeline + no-answer threshold for SQuAD v2
nli, classification	eval_nli.py	Label remapping, filter label=-1
zero_shot	eval_zero_shot.py	Pipeline + candidate_labels
multilabel	eval_multilabel.py	Single-label argmax for accuracy metric
summarization	eval_summarization.py	Seq2Seq + ROUGE
ner	eval_ner.py	Token classification + seqeval, parquet fallback
vision	eval_vision.py	AutoImageProcessor, grayscale→RGB
sentiment	eval_sentiment.py	Standard classification
multiple_choice	eval_multiple_choice.py	vLLM + conditional log-likelihood

How to use

Read the matching template: cat scripts/eval_<type>.py
Adapt the CONFIG section at top (MODEL_ID, DATASET_ID, SPLIT, etc.)
Run via run_code_in_docker