paper2skill-evaluation-infrastructure

name

paper2skill-evaluation-infrastructure

description

Convert dataset and benchmark papers into evaluation infrastructure skills. For datasets: extracts collection protocol, annotation design, quality control. For benchmarks: extracts task definition, metric selection, leaderboard design. Use this skill when extracting skills from Category 2 (Evaluation Infrastructure) papers — ImageNet-style dataset papers, SWE-bench-style benchmark papers, or any paper whose primary contribution is evaluation methodology.

When to Use

Apply this skill when you encounter arXiv papers that:

• Introduce a new dataset with annotation/collection procedures (ImageNet, COCO, GLUE, SWE-bench)
• Define a new benchmark or evaluation protocol with metrics justification
• Describe annotation schema, inter-annotator agreement, or quality assurance procedures
• Discuss scale considerations or data curation decisions
• Analyze what a benchmark actually measures versus assumptions people make about it

Examples: COCO dataset design, GLUE benchmark construction, SWE-bench evaluation framework, SafetyBench methodology.

When NOT to Use

Do not use this skill for:

• Papers that only USE a dataset without describing its construction
• Purely methodological papers (even if they include evaluation)
• Papers that evaluate on benchmarks without describing benchmark design
• Survey papers that analyze existing benchmarks
• Papers about data augmentation without collection/annotation focus

---

Extraction Template

For Dataset Papers

Step 1: Collection Methodology

Extract how the raw data was gathered and organized.

**Collection Source:** Where does the raw data come from?
**Collection Scale:** How much data was collected? By whom?
**Collection Process:**
- Sampling strategy (random, stratified, targeted)
- Data preparation steps
- Format standardization
**Scale Considerations:** Storage, accessibility, versioning

Step 2: Annotation Design

Document the human annotation or labeling process.

**Annotation Schema:** What exactly is being annotated?
**Annotation Protocol:**
- Step-by-step instructions for annotators
- Definitions and edge cases
- Examples (prototypical and boundary cases)
**Quality Control:**
- Inter-annotator agreement metrics
- Agreement resolution procedure
- Crowd-worker vetting/training
**Annotation Cost:** Time and resource estimates

Step 3: Data Format & Specs

Extract data structure and usage specifications.

**Data Format:** JSON/CSV/HDF5 structure
**Splits:** Train/val/test reasoning and sizes
**Metadata:** What additional information is provided?
**Access & Licensing:** How is the dataset distributed?

For Benchmark Papers

Step 1: Task Definition

Extract what the benchmark actually measures.

**Task Definition:** What is the benchmark asking models to do?
**Input/Output Spec:**
- Input format and constraints
- Output format and constraints
- Assumptions about model architecture or training
**Task Motivation:** Why is this task important?
**Gap from Real World:** What simplifications are made?

Step 2: Evaluation Metrics

Document metric selection and interpretation.

**Primary Metrics:** [Metric name, formula, range]
**Why These Metrics:** What capability does each metric isolate?
**Metric Limitations:** What does each metric NOT measure?
**Secondary Metrics:** [How they provide additional insight]
**Baseline Selection:** What baselines are included and why?

Step 3: Benchmark Analysis

Extract what the benchmark reveals.

**What It Measures:** Core capabilities tested
**What People Think It Measures:** Common misconceptions
**Failure Modes:** What types of errors does it NOT catch?
**Correlation with Real Performance:** If studied, how well does benchmark performance predict real-world success?
**Known Saturation:** Is the benchmark being saturated? What's next?

Step 4: Leaderboard Design

For competitive benchmarks, extract evaluation infrastructure.

**Leaderboard Mechanics:** How is performance tracked?
**Submission Process:** Frequency, format, evaluation turnaround
**Evaluation Server:** How are answers checked?
**Cheating Prevention:** How is overfitting to eval set prevented?
**Reporting Standards:** What metadata must submissions include?

---

Output Skill Format

Generate a new SKILL.md with:

Frontmatter:

---
name: [kebab-case-dataset-or-benchmark-name]
title: [Dataset/Benchmark: {Name} Evaluation Infrastructure]
version: 0.0.2
engine: skillxiv-v0.0.2-claude-opus-4.6
license: MIT
url: [verified arxiv link to source paper]
keywords: [evaluation, infrastructure, metric-selection, annotation-protocol, or data-curation]
description: Build evaluation infrastructure for {domain} by replicating {benchmark/dataset} methodology. Includes {annotation protocol OR task definition}, {quality control OR metric selection}, enabling {outcome} for {use }

For Dataset Skills, include:

1. Collection Methodology (1 paragraph): How raw data is gathered
2. Annotation Protocol (3-4 bullet points): Step-by-step labeling procedure
3. Quality Control (2-3 bullet points): Inter-annotator agreement and resolution
4. Data Format (1 paragraph): Structure and splits
5. When to Use (1-2 sentences): What research uses this dataset type

For Benchmark Skills, include:

1. Task Definition (1 paragraph): What the benchmark measures
2. Evaluation Metrics (3-5 bullet points): Metric selection rationale
3. What It Actually Measures (1 paragraph): Reality vs perception
4. Baseline Strategy (2-3 bullet points): How to select and interpret baselines
5. When to Use (1-2 sentences): What research questions this benchmark answers

Length: 150-250 lines

---

Processing Instructions

1. Obtain the paper: Fetch HTML from arxiv.org/html/{arxiv_id}, fallback to PDF
2. Classify: Determine if this is a dataset paper (annotation focus) or benchmark paper (task/metric focus)
3. Extract methodology: For datasets, extract collection and annotation details. For benchmarks, extract task and metric definitions.
4. Analyze depth: Determine if paper provides sufficient methodological detail (some papers lack specifics)
5. Identify gaps: What assumptions does the benchmark/dataset make? What real-world aspects are simplified?
6. Validate: Confirm the paper contains original evaluation infrastructure (not just application)
7. Generate skill: Write output skill following the appropriate format above

---

Quality Checks

• Paper clearly describes collection/annotation OR task/metric methodology
• Specific procedures are documented (not just high-level overview)
• Quality control or metric selection rationale is explained
• Output skill identifies what the benchmark actually measures vs common assumptions
• For datasets: annotation protocol is detailed enough to replicate
• For benchmarks: metrics are justified, not arbitrary
• Keywords (5-10) include "evaluation", "infrastructure", and domain-specific terms
• Description is under 1024 characters
• Engine tag matches skillxiv-v0.0.2-claude-opus-4.6

---

Common Pitfalls

• Assuming all dataset papers are extractable: Skip papers that only distribute existing data
• Mistaking metric innovation for benchmark innovation: Paper may introduce a metric without designing the benchmark itself
• Missing the gap: If the paper doesn't explain WHY these design choices, extract them anyway with domain reasoning
• Oversimplifying complexity: Real annotation protocols are complex; preserve the details
• Ignoring leaderboard design: Leaderboard mechanics are part of the benchmark infrastructure