eval-templates

Estrellas38

Forks3

Actualizado12 de mayo de 2026 a las 23:06

Runnable evaluation template scripts for ML tasks. Match task_type to template, adapt CONFIG, run.

Instalación

Instalar con Codex o Claude Copia este prompt, pégalo en Codex, Claude u otro asistente, y deja que revise la página de la skill y la instale por ti.

Ejecutar en Manus

Fuente

allenai

allenai/artifact-linker

Abrir repositorio de GitHub Ver repositorios del creador

Descarga

Ejecutar en Manus

Ocupaciones relacionadasSOC

Basado en la clasificación ocupacional SOC

Científicos de datosOcupaciones informáticas y matemáticas·SOC 15-2051

Explorador de archivos

10 archivos

SKILL.md

readonly

name	eval-templates
description	Runnable evaluation template scripts for ML tasks. Match task_type to template, adapt CONFIG, run.

Evaluation Templates

Match the plan's task_type to a template in scripts/, adapt CONFIG variables, run via run_code_in_docker.

task_type	Script	Notes
extractive_qa	eval_qa.py	Pipeline + no-answer threshold for SQuAD v2
nli, classification	eval_nli.py	Label remapping, filter label=-1
zero_shot	eval_zero_shot.py	Pipeline + candidate_labels
multilabel	eval_multilabel.py	Single-label argmax for accuracy metric
summarization	eval_summarization.py	Seq2Seq + ROUGE
ner	eval_ner.py	Token classification + seqeval, parquet fallback
vision	eval_vision.py	AutoImageProcessor, grayscale→RGB
sentiment	eval_sentiment.py	Standard classification
multiple_choice	eval_multiple_choice.py	vLLM + conditional log-likelihood

How to use

Read the matching template: cat scripts/eval_<type>.py
Adapt the CONFIG section at top (MODEL_ID, DATASET_ID, SPLIT, etc.)
Run via run_code_in_docker

Más de este repositorio

mismo repositorio

hugging-face-dataset-viewer

allenai/artifact-linker

Use this skill for Hugging Face Dataset Viewer API workflows that fetch subset/split metadata, paginate rows, search text, apply filters, download parquet URLs, and read size or statistics.

2026-05-1238

hugging-face-datasets

allenai/artifact-linker

Create and manage datasets on Hugging Face Hub. Supports initializing repos, defining configs/system prompts, streaming row updates, and SQL-based dataset querying/transformation. Designed to work alongside HF MCP server for comprehensive dataset workflows.

2026-05-1238

hugging-face-evaluation

allenai/artifact-linker

Add and manage evaluation results in Hugging Face model cards. Supports extracting eval tables from README content, importing scores from Artificial Analysis API, and running custom model evaluations with vLLM/lighteval. Works with the model-index metadata format.

2026-05-1238

name	eval-templates
description	Runnable evaluation template scripts for ML tasks. Match task_type to template, adapt CONFIG, run.

Evaluation Templates

Match the plan's task_type to a template in scripts/, adapt CONFIG variables, run via run_code_in_docker.

task_type	Script	Notes
extractive_qa	eval_qa.py	Pipeline + no-answer threshold for SQuAD v2
nli, classification	eval_nli.py	Label remapping, filter label=-1
zero_shot	eval_zero_shot.py	Pipeline + candidate_labels
multilabel	eval_multilabel.py	Single-label argmax for accuracy metric
summarization	eval_summarization.py	Seq2Seq + ROUGE
ner	eval_ner.py	Token classification + seqeval, parquet fallback
vision	eval_vision.py	AutoImageProcessor, grayscale→RGB
sentiment	eval_sentiment.py	Standard classification
multiple_choice	eval_multiple_choice.py	vLLM + conditional log-likelihood

How to use

Read the matching template: cat scripts/eval_<type>.py
Adapt the CONFIG section at top (MODEL_ID, DATASET_ID, SPLIT, etc.)
Run via run_code_in_docker