eval-templates

Sterne38

Forks3

Aktualisiert12. Mai 2026 um 23:06

Runnable evaluation template scripts for ML tasks. Match task_type to template, adapt CONFIG, run.

Installation

Mit Codex oder Claude installieren Kopieren Sie diesen Prompt, fügen Sie ihn in Codex, Claude oder einen anderen Assistant ein und lassen Sie die Skill-Seite prüfen und installieren.

In Manus ausführen

Quelle

allenai

allenai/artifact-linker

GitHub-Repository öffnen Creator-Repositorys ansehen

Download

In Manus ausführen

Verwandte BerufeSOC

Basierend auf der SOC-Berufsklassifikation

DatenwissenschaftlerInformatik- und Mathematikberufe·SOC 15-2051

Datei-Explorer

10 Dateien

SKILL.md

readonly

name	eval-templates
description	Runnable evaluation template scripts for ML tasks. Match task_type to template, adapt CONFIG, run.

Evaluation Templates

Match the plan's task_type to a template in scripts/, adapt CONFIG variables, run via run_code_in_docker.

task_type	Script	Notes
extractive_qa	eval_qa.py	Pipeline + no-answer threshold for SQuAD v2
nli, classification	eval_nli.py	Label remapping, filter label=-1
zero_shot	eval_zero_shot.py	Pipeline + candidate_labels
multilabel	eval_multilabel.py	Single-label argmax for accuracy metric
summarization	eval_summarization.py	Seq2Seq + ROUGE
ner	eval_ner.py	Token classification + seqeval, parquet fallback
vision	eval_vision.py	AutoImageProcessor, grayscale→RGB
sentiment	eval_sentiment.py	Standard classification
multiple_choice	eval_multiple_choice.py	vLLM + conditional log-likelihood

How to use

Read the matching template: cat scripts/eval_<type>.py
Adapt the CONFIG section at top (MODEL_ID, DATASET_ID, SPLIT, etc.)
Run via run_code_in_docker

Mehr aus diesem Repository

gleiches Repository

hugging-face-dataset-viewer

allenai/artifact-linker

Use this skill for Hugging Face Dataset Viewer API workflows that fetch subset/split metadata, paginate rows, search text, apply filters, download parquet URLs, and read size or statistics.

2026-05-1238

hugging-face-datasets

allenai/artifact-linker

Create and manage datasets on Hugging Face Hub. Supports initializing repos, defining configs/system prompts, streaming row updates, and SQL-based dataset querying/transformation. Designed to work alongside HF MCP server for comprehensive dataset workflows.

2026-05-1238

hugging-face-evaluation

allenai/artifact-linker

Add and manage evaluation results in Hugging Face model cards. Supports extracting eval tables from README content, importing scores from Artificial Analysis API, and running custom model evaluations with vLLM/lighteval. Works with the model-index metadata format.

2026-05-1238

name	eval-templates
description	Runnable evaluation template scripts for ML tasks. Match task_type to template, adapt CONFIG, run.

Evaluation Templates

Match the plan's task_type to a template in scripts/, adapt CONFIG variables, run via run_code_in_docker.

task_type	Script	Notes
extractive_qa	eval_qa.py	Pipeline + no-answer threshold for SQuAD v2
nli, classification	eval_nli.py	Label remapping, filter label=-1
zero_shot	eval_zero_shot.py	Pipeline + candidate_labels
multilabel	eval_multilabel.py	Single-label argmax for accuracy metric
summarization	eval_summarization.py	Seq2Seq + ROUGE
ner	eval_ner.py	Token classification + seqeval, parquet fallback
vision	eval_vision.py	AutoImageProcessor, grayscale→RGB
sentiment	eval_sentiment.py	Standard classification
multiple_choice	eval_multiple_choice.py	vLLM + conditional log-likelihood

How to use

Read the matching template: cat scripts/eval_<type>.py
Adapt the CONFIG section at top (MODEL_ID, DATASET_ID, SPLIT, etc.)
Run via run_code_in_docker