create-eval

name	create-eval
description	Implement a new evaluation from an issue or paper. Guides through requirements gathering, architecture design, scaffolding, implementation, testing, and quality checks. Use when user asks to create/implement/build a new evaluation.

Create Evaluation

Implement a new evaluation from an issue, paper, or benchmark specification. This is a phased workflow with human checkpoints between major phases. You should usually do human checkpoints. If the user requests that you do not do this, or you are in a fully autonomous workflow that says that there is no user to check with, you should ignore the human checkpoints. If you check in with the user several times and the user always replies with the same generic message like "Please continue the task", this indicates you are in an autonomous workflow.

Expected Arguments

When invoked, this skill expects:

• issue (optional): A GitHub issue number or URL (e.g., #42, https://github.com/<owner>/<repo>/issues/42)
• eval_name (optional): The name for the evaluation directory (e.g., frontier_cs)
• paper_url (optional): URL to the paper (arXiv, etc.)
• reference_impl (optional): URL to the reference implementation

If not provided, ask the user. At minimum, the eval_name is required.

Phase 1: Requirements Gathering

Collect all information needed to make architectural decisions.

1.1 Gather Sources

• If an issue was provided, fetch it with gh issue view <number> --json title,body,labels
• If a paper URL was provided, fetch it with WebFetch
• If a reference implementation was provided, explore its structure
• If a HuggingFace dataset exists, fetch its metadata: https://huggingface.co/api/datasets/<name>

1.2 Classify the Evaluation

Determine the eval type by answering these questions:

Question	Options
What does the model produce?	Text answer / Code / Multi-step actions / Structured output
How is it scored?	Exact match / Partial credit / LLM judge / Code execution / Custom checker
Does scoring need a sandbox?	No / Yes (Docker)
Does the solver need tools?	No / Yes (bash, python, custom)
Is it multi-turn?	No / Yes
What's the dataset source?	HuggingFace / GitHub / URL / Local

Use this decision tree to pick the eval archetype:

Model produces text answer?
  Scored by exact match? ──> Simple Q&A (pattern: gpqa, gsm8k)
  Scored by LLM judge? ──> LLM-as-judge (pattern: mt_bench)
  Scored by code execution? ──> Code-in-sandbox (pattern: humaneval)

Model produces code?
  Single file, run + check? ──> Code generation (pattern: humaneval)
  Multi-step with tools? ──> Agentic coding (pattern: swe_bench)

Model takes multi-step actions?
  With bash/python tools? ──> Agentic (pattern: swe_bench, agent_bench)
  With custom tools? ──> Agentic + custom tools (pattern: gdm_intercode_ctf)

1.3 Identify Key Components

Document:

• Dataset: Source, format, columns, splits, sample count, license
• Scoring: Mechanism, metrics (binary vs continuous), partial credit rules
• Sandbox needs: What needs to run in Docker, what software must be installed
• Dependencies: Any Python packages needed beyond inspect_ai
• Task variants: Multiple tasks, parameterized variants, or single task

1.4 Write Requirements

Create agent_artefacts/<eval_name>/create/REQUIREMENTS.md with all findings.

1.5 Human Checkpoint

Present a summary to the user and wait for approval before proceeding. Include:

• Eval archetype classification
• Proposed task structure (how many @task functions, what parameters)
• Scoring approach
• Sandbox needs (if any)
• Any open questions or ambiguities

Phase 2: Architecture Design

2.1 Select Reference Eval(s)

Choose 1-2 existing evals as architectural models based on the archetype:

Archetype	Reference Eval	Key Pattern
Simple Q&A	gpqa	hf_dataset + built-in scorer
Math with few-shot	gsm8k	Few-shot prompting + match(numeric=True)
Code generation	humaneval	generate() solver + custom @scorer with sandbox
Agentic	swe_bench	basic_agent() solver + bash()/python() tools + Docker
LLM-as-judge	mt_bench	model_graded_qa() or custom judge scorer

Read the reference eval's source code to understand its patterns.

2.2 Design File Structure

Determine what files are needed:

Always required:

src/<eval_name>/
  __init__.py           # Public exports
  <eval_name>.py        # @task function(s), solver, scorer (if simple)
  eval.yaml             # Metadata
  README.md             # Documentation

Add when needed:

  dataset.py            # When dataset loading is complex (multi-source, caching, transforms)
  scorers.py            # When scorer has significant custom logic
  prompts.py            # When there are multiple/complex prompt templates
  compose.yaml          # When Docker sandbox is needed
  Dockerfile            # When a custom Docker image is needed

Always required for tests:

tests/<eval_name>/
  __init__.py
  test_<eval_name>.py   # Unit tests + E2E (or split into test_dataset.py, test_scorer.py, test_e2e.py)

2.3 Design Dataset Loading

Choose the dataset loading strategy:

HuggingFace dataset?
  Simple columns? ──> hf_dataset() with FieldSpec or record_to_sample
  Complex transforms? ──> hf_dataset() with record_to_sample function

GitHub-hosted data?
  Small, static? ──> Download to INSPECT_EVALS_CACHE_PATH at task init
  Large or versioned? ──> Download + cache with revision pinning

Local data?
  JSON? ──> json_dataset()
  CSV? ──> csv_dataset()

Always pin dataset versions:

• HuggingFace: Use revision parameter with commit SHA from https://huggingface.co/api/datasets/<name> (sha field)
• GitHub: Pin to a specific commit SHA

2.4 Design Scoring

Choose the scoring strategy:

Binary outcome (correct/incorrect)?
  ──> Score.value = CORRECT/INCORRECT, metrics = [accuracy(), stderr()]

Continuous score (0.0-1.0 or 0-100)?
  ──> Score.value = float, metrics = [mean(), stderr()]
  ──> Normalize to 0.0-1.0 range for Score.value

Multiple sub-scores?
  ──> Score.value = dict, custom metrics or per-key mean()/stderr()

Key rules from BEST_PRACTICES.md:

• Store all aggregatable values in Score.value, NOT in Score.metadata
• Use Score.metadata only for identifiers and debugging info
• Use Score.explanation for human-readable scoring details

2.5 Design Solver

Simple generation?
  ──> generate() or [prompt_template(...), generate()]

Needs system message?
  ──> [system_message(...), generate()]

Needs tools (bash, python)?
  ──> basic_agent(init=system_message(...), tools=[bash(), python()])

Needs few-shot examples?
  ──> [system_message(fewshot_text), prompt_template(...), generate()]

Paper methodology check: If the original paper uses single-turn evaluation but an agentic approach is also valid, implement both solvers and expose a parameter (e.g., agentic: bool = True). Default to agentic (since this is an agentic framework) but allow paper-faithful reproduction with agentic=False. Include separate system prompt constants for each mode.

2.5b Identify Hardware/Environment Requirements

Check if any subset of problems requires special hardware (GPUs, large memory, specific software). If so:

• Add a filtering parameter (e.g., include_gpu_problems: bool = False) that defaults to excluding problems the user likely can't run
• Document which problems are excluded and why
• Tag problems in metadata so filtering is data-driven, not hardcoded to specific IDs

This applies broadly: GPU-dependent problems, problems needing internet access, problems requiring specific commercial software, etc.

2.6 Write Architecture

Create agent_artefacts/<eval_name>/create/ARCHITECTURE.md with:

• File structure with descriptions
• Dataset loading approach
• Scoring mechanism
• Solver design
• Docker setup (if applicable)
• Dependencies needed

2.7 Human Checkpoint

Present the architecture to the user and wait for approval.

Phase 3: Scaffolding

3.1 Create Directories

mkdir -p src/<eval_name>
mkdir -p tests/<eval_name>

3.2 Create eval.yaml

title: "Full Title: Subtitle"
description: |
  Description from the paper abstract.
arxiv: https://arxiv.org/abs/XXXX.XXXXX
group: <Group>  # One of: Assistants, Coding, Cybersecurity, Knowledge, Mathematics, Multimodal, Reasoning, Safeguards, Scheming, Writing
contributors: ["github-username"]
version: "1-A"
tasks:
  - name: <task_name>
    dataset_samples: <count>
tags: ["Agent"]  # Only if agentic. Omit if not applicable.
metadata:
  sandbox: ["solver", "scorer"]  # Only if sandbox used. Omit individual items or entire field if not needed.

3.3 Create init.py

Export all public symbols: task functions, dataset functions, scorer functions, constants.

3.4 Create Docker Files (if needed)

compose.yaml (at eval root, NOT in data/):

services:
  default:
    build: .
    init: true
    command: tail -f /dev/null

Dockerfile:

• Start from a minimal base (e.g., ubuntu:24.04, python:3.12-slim)
• Install only what's needed for scoring/execution
• Keep it small — models don't need IDE tools

3.5 Create Test Stubs

Create tests/<eval_name>/__init__.py (empty).

3.6 Pin Dataset Version

• For HuggingFace: Fetch SHA from API and use as revision parameter
• For GitHub: Get commit SHA and pin downloads to it

3.7 Add Optional Dependencies (if needed)

Add to pyproject.toml:

[project.optional-dependencies]
<eval_name> = ["package1>=1.0", "package2"]

# Also add to your test extras so tests have access:
test = [
  # ... existing entries ...
  "<your-package-name>[<eval_name>]",  # name from [project] in pyproject.toml
]

Critical rule: Optional dependencies must NOT be imported at module level. Defer imports to function scope.

3.8 Run Linting

make check

This runs ruff format, ruff check, mypy, README generation, and markdownlint in one command.

Phase 4: Core Implementation

Implement in dependency order: dataset -> prompts -> scorer -> solver -> @task.

4.1 Dataset Loading

• Use hf_dataset() from inspect_ai.dataset for HuggingFace datasets
• Write a record_to_sample function that maps raw records to Sample objects
• Include all useful fields in sample.metadata for debugging
• Use stable, canonical IDs for sample.id

4.2 Prompts

• Keep prompt text as module-level constants (ALL_CAPS)
• Use .format() or f-strings for variable injection
• Separate system prompt from user prompt

4.3 Scorer

• Prefer built-in scorers (match(), accuracy(), model_graded_qa()) when possible
• For custom scorers, use @scorer(metrics=[...]) decorator
• For sandbox execution in scorer, use await sandbox().exec(...)
• Handle timeouts and errors gracefully
• Return Score(value=..., answer=..., explanation=...)

4.4 Solver

• Prefer built-in solvers (generate(), basic_agent())
• For agentic evals, use basic_agent(init=system_message(...), tools=[...])
• Set sensible message_limit defaults

4.5 Task Function

EVAL_VERSION = load_eval_metadata("<eval_name>").version

@task
def <eval_name>(...) -> Task:
    return Task(
        dataset=...,
        solver=...,
        scorer=...,
        sandbox=...,  # "docker" if needed
        version=EVAL_VERSION.comparability_version,
        metadata=EVAL_VERSION.to_metadata(),
    )

Key rules:

• Do NOT pass a name parameter to @task
• Use load_eval_metadata for version info
• Use absolute imports throughout

4.6 Register the Task

Expose your @task functions via the inspect_ai entry-point group in pyproject.toml so the inspect eval CLI can find them:

[project.entry-points.inspect_ai]
<eval_name> = "<eval_name>"

The key is the prefix the inspect eval CLI will use; the value is the Python module that exports your @task functions (re-export them from the package's __init__.py).

4.7 Run Linting

make check

4.8 Human Checkpoint

Tell the user the core implementation is done. Suggest they smoke-test:

uv run inspect eval <eval_name>/<task_name> --model mockllm/model --limit 1

For Docker-based evals, this requires Docker to be running. If the user can't run Docker, note this and move on.

Phase 5: Testing

5.1 Create Tests

Follow the patterns from the ensure-test-coverage skill. At minimum:

record_to_sample test (use real data from the dataset):

def test_record_to_sample() -> None:
    record = { ... }  # Real example from dataset
    sample = record_to_sample(record)
    assert sample.id == "expected_id"
    assert "expected_text" in str(sample.input)
    assert sample.target == "expected_target"

HuggingFace dataset structure validation (if using HF datasets):

Use assert_huggingface_dataset_structure from src/utils/huggingface.py to document and validate the expected schema:

from utils.huggingface import (
    DatasetInfosDict,
    assert_huggingface_dataset_structure,
    get_dataset_infos_dict,
)

@pytest.fixture(scope="module")
def dataset_infos() -> DatasetInfosDict:
    return get_dataset_infos_dict(HF_DATASET_PATH)

@pytest.mark.huggingface
def test_dataset_structure(dataset_infos: DatasetInfosDict) -> None:
    assert_huggingface_dataset_structure(
        dataset_infos,
        {"configs": {"default": {"splits": ["test"], "features": {...}}}},
    )

E2E test with mockllm:

@pytest.mark.dataset_download  # If downloads data
@pytest.mark.docker            # If uses Docker
def test_<eval_name>_e2e() -> None:
    [log] = eval(
        tasks=<task_function>(),
        model="mockllm/model",
        limit=1,
        message_limit=1,  # For agentic evals
    )
    assert log.status == "success"

Custom scorer/solver/tool tests if any exist. At minimum a type-check (isinstance(scorer, Scorer)) is required by autolint for custom scorers.

5.2 Add Pytest Markers

• @pytest.mark.dataset_download — if test downloads a dataset
• @pytest.mark.huggingface — if test uses a HuggingFace dataset
• @pytest.mark.docker — if test uses Docker sandbox
• @pytest.mark.slow(<seconds>) — if test takes >10 seconds (use observed duration)

5.3 Run Tests

uv run pytest tests/<eval_name>/ -v

Fix any failures. For tests requiring Docker or datasets, note which tests need special setup.

Phase 6: Quality and Submission

6.1 Run Autolint

uv run python tools/run_autolint.py <eval_name>

The autolint surfaces the registry's recommended structural standards. In this template they are advisory by default — fix what you can but failures won't block the build unless ENFORCE_AUTOLINT=true is set in tools/enforcement.config. See README.md "Checks and enforcement".

6.2 Run All Linting

make check

This single command runs ruff format, ruff check, mypy, README generation, and markdownlint. Fix any errors before proceeding.

6.3 Generate README (if needed separately)

uv run python tools/generate_readmes.py --create-missing-readmes

Note: This tool does NOT accept eval name as argument — it discovers and generates all missing READMEs. Then fill in any TODO sections that are generated (description paragraph, dataset section, scoring section). The evaluation report TODO can be left for later.

6.4 Verify Against EVALUATION_CHECKLIST.md

Read EVALUATION_CHECKLIST.md and verify every applicable item. Key information not in EVALUATION_CHECKLIST.md

Smoke Test: Ensuring the eval runs with --limit 1 on a real model.

If the evaluation takes longer than five minutes to run and you can identify what component is failing, Add tests to reproduce any failure modes first. Only once those pass should you run it again.

Evaluation Report: This is the place you will stop for human approval unless you are specifically told not to or it is clear you are in a fully autonomous workflow.

Trajectory Analysis: If there are significant problems in trajectory analysis, these should be fixed. If the problem is related to the code at all, such as a broken environment, you should fix them using Test Driven Development. Create a unit test or tests with data that reproduces the failure mode, verify the test fails, then fix the issue and verify the test succeeds.

Gotchas (Lessons from Real Implementations)

• Regex with language names: If building regex patterns from language names like c++, use re.escape() — the + is a regex special character.
• Filtering datasets by category: hf_dataset() with FieldSpec loads all samples. If you need to filter (e.g., by category/track), load once, convert to samples, then filter. Use MemoryDataset(samples=filtered_list, name=...) to create a new dataset (NOT Dataset() which takes no constructor args).
• Mixed config formats: Dataset fields that look like YAML may contain JSON in some records. Always implement fallback parsing (YAML → JSON → raw string).
• basic_agent() API: Use basic_agent(init=system_message(...), tools=[...]). This is the preferred agentic solver. Only use react() if a more complex, bespoke solver is required for a given evaluation.
• sandbox().exec() API: The parameter for passing stdin is input=, NOT stdin=. Signature: exec(cmd, input=None, cwd=None, env=None, user=None, timeout=None).
• HF dataset structure validation: Use assert_huggingface_dataset_structure from src/utils/huggingface.py (from utils.huggingface import ...) to document the expected schema. This validates field names/types against what the loader produces, so a silent upstream schema change is caught early.
• README generator: tools/generate_readmes.py --create-missing-readmes — does NOT accept eval name arg, generates all missing READMEs at once.
• Linting: Use make check to run all linting (ruff format, ruff check, mypy, README generation, markdownlint). Do NOT run individual linting commands — make check covers everything.
• Scorer robustness for agentic evals: When a model hits message_limit during a tool call, state.output.completion is empty. If your scorer extracts code from the completion, add a fallback that reads solution files from the sandbox (e.g., find /tmp /home -name '*.cpp').
• Version pinning in Dockerfiles: Pin all downloaded dependencies (e.g., testlib.h) to specific commit SHAs, not branch names like master or main. Same applies to GitHub archive downloads in dataset code.
• Paper vs agentic methodology: If a paper uses single-turn evaluation, implement both single-turn and agentic solvers. Default to agentic, but expose a parameter so users can reproduce paper results. Compare against the reference implementation's system prompts.
• Hardware-gated problems: Some benchmarks include problems requiring GPUs, specific hardware, or commercial software. Add a filtering parameter (e.g., include_gpu_problems=False) so these are excluded by default but can be opted into.
• Multiple execution modes in one benchmark: Some benchmarks have problems with fundamentally different execution models (e.g., batch file I/O vs interactive bidirectional piping). Check the reference implementation's config schema for type fields and implement all modes, not just the most common one.
• Sandbox config must reference compose.yaml: Passing sandbox="docker" uses a bare container. To use a custom Dockerfile, pass sandbox=("docker", str(compose_file)) where compose_file = Path(__file__).parent / "compose.yaml".
• HF image fields → base64 data URI: When loading a HuggingFace dataset that contains images, hf_dataset() delivers them as {"bytes": ..., "path": ...} dicts. Encode inline: image_uri = f"data:image/jpeg;base64,{base64.b64encode(image_bytes).decode()}" and pass to ContentImage(image=image_uri). No disk cache or PIL thumbnail needed — hf_dataset() already caches the dataset locally.
• Dataset.filter() return value must be assigned: dataset.filter(pred) returns a new dataset; it does not mutate in place. dataset.filter(...) on a line by itself silently does nothing. Always write dataset = dataset.filter(...).
• Use create_stable_id for sample IDs: from utils import create_stable_id and call create_stable_id(*fields, prefix=..., length=8) rather than computing raw hashlib.md5(...).hexdigest() yourself. It uses a null-byte delimiter between fields so combinations like ("ab", "c") and ("a", "bc") hash differently.
• Include image content in sample IDs: When a dataset contains image fields and the same question can be asked about multiple different images (common in VQA benchmarks), the sample ID must include a hash of the image bytes, not just the question text. Hashing question text alone produces duplicate IDs.
• Check answer length distribution before committing to exact match: Before implementing a scorer for open-ended answers, inspect the distribution of ground-truth answer lengths (Counter(len(a.split()) for a in answers)). Long multi-word answers are practically unreproducible verbatim — consider filtering out long answers with a MAX_ANSWER_WORDS constant and update eval.yaml dataset_samples to the post-filter count. Alternatively, switch to a model graded scorer.
• Constrain binary-answer format explicitly: For yes/no questions, the prompt must say "answer ONLY 'yes' or 'no'" (or equivalent). Without this constraint, models append qualifications ("Yes, there is cardiomegaly") which fail exact match even when the answer is correct.

Things NOT to Do

• Don't pass name to @task: This is only for dynamically created tasks
• Don't import optional dependencies at module level: Defer to function scope
• Don't use relative imports: Use absolute imports (e.g. from <your_package>.<module> import <symbol>) so modules and tests run from any working directory
• Don't use private Inspect AI APIs: Avoid inspect_ai.*._internal modules
• Don't store aggregatable values in Score.metadata: Use Score.value (can be a dict)
• Don't upload new dataset copies: Use official sources
• Don't implement manual HF token checks: The centralized conftest handles @pytest.mark.huggingface
• Don't over-test thin wrappers: isinstance(solver, Solver) is sufficient for basic_agent wrappers
• Don't add type annotations that are redundant: Only annotate function signatures
• Don't create documentation files unless asked: The README is created by generate_readmes.py
• Don't run uv update: Only uv lock if dependencies change
• Don't hardcode magic numbers: Extract to named constants if used 3+ times or unclear
• Don't write try/catch unless truly necessary: Let errors crash early