eval-validity-review

name	eval-validity-review
description	Review a single evaluation's validity — whether its claims hold up, whether its name is accurate, whether samples can be both succeeded and failed at, and whether scoring measures ground truth. Use when user asks to check validity of an eval, or as part of the Master Checklist workflow. Do NOT use for code quality or test coverage (use eval-quality-workflow or ensure-test-coverage instead).

Evaluation Validity Review

This skill assesses whether an evaluation measures what it claims to measure. It checks four dimensions of validity:

1. Claims Coherence — Do the evaluation's claims about its data, mechanisms, capabilities, and results hold up under scrutiny?
2. Name Validity — Does the name accurately represent the capability being measured?
3. Dataset Validity — Can models both succeed and fail at each sample given the available affordances?
4. Scoring Validity — Does the scorer measure ground truth rather than proxies?

The first dimension is a high-level sense check that should be performed before diving into the mechanical details of the other three. If the evaluation's claims are fundamentally false or unsupported, the detailed checks matter less.

This review is read-only. It produces a report with findings and recommendations but does not modify evaluation code.

Important: The phases below are guidelines, not a rigid checklist. Use your judgement throughout. If something feels off at any point — an explanation that doesn't quite add up, a mechanism that seems more complex than it needs to be, samples that feel generic rather than sourced — investigate it, even if no specific step tells you to. The goal is to determine whether the evaluation is sound, not to mechanically complete every listed check.

Identifying the Evaluation

If the user has given you a name, that takes priority. If you were just building an evaluation, or the user has uncommitted code for one specific evaluation, you can assume that's the correct one. If you are not confident which evaluation to look at, ask the user.

Setup

1. Determine the evaluation name (the folder name under src/). This will be referred to as <eval_name>.
2. Determine the task version from the @task function's version argument. Combine them as <eval_name>_<version> (e.g., gpqa_1_1_2). If this exact folder name already exists under agent_artefacts/, append a number (e.g., gpqa_1_1_2_2). This combined name is referred to as <artefact_name>.
3. Create the output directory: agent_artefacts/<artefact_name>/validity/
4. All .md files created by this workflow go in that directory.
5. Create a NOTES.md file for recording observations during the review. Err on the side of taking lots of notes.

Phase 1: Gather Context

Before making any judgements, read and record key facts about the evaluation. You will need all of the following:

1.1 Task Definition

Read the main task file(s) in src/<eval_name>/. Record:

• All @task function names and their parameters
• The task version
• What solver is used (simple solver like multiple_choice(), or an agent like basic_agent())
• What scorer is used
• What tools are provided to the agent (if any)
• Whether a sandbox is configured (look for sandbox= in the Task constructor and compose.yaml files)
• The system message or prompt template given to the model

1.2 Dataset and Samples

Read the dataset loading code. Record:

• Where the data comes from (HuggingFace, CSV, JSON, inline)
• The record_to_sample function or equivalent transformation
• What fields are in each Sample: input, target, choices, metadata, sandbox
• If the dataset is small or inline, read the actual samples. If it is large, read a representative subset by examining the transformation function and any available example data in the README.

1.3 Scorer

Read the scorer implementation (either in the eval's own files or identify which Inspect built-in is used). Record:

• What the scorer checks (exact match, substring, LLM judge, sandbox state, custom logic)
• What constitutes a score of 1 (success) vs 0 (failure)
• Whether the scorer uses an LLM judge, and if so, what prompt it uses
• Whether the scorer checks the actual outcome or a proxy for the outcome

1.4 README and eval.yaml

Read the README and eval.yaml. Record:

• The evaluation's stated purpose and description
• The group field in eval.yaml (e.g., "Reasoning", "Safeguards", "Coding")
• Any referenced paper or source material (arxiv link, paper title)

1.5 Sandbox Environment (if applicable)

If the evaluation uses a Docker sandbox, read the compose.yaml and any setup scripts. Record:

• What services are running
• What files or resources are available to the agent inside the sandbox
• Network access (is the internet available?)
• Any pre-configured state (databases, servers, files with specific content)

1.6 Dataset Quality Signals

While reading the dataset (whether YAML, JSON, CSV, or HuggingFace), note any quality concerns:

• Placeholder or stub content: Prompts that reference "lengthy content" or "2000 tokens of context" without actually including it. Fields that are empty or contain generic filler where real content is expected.
• Unsubstantiated claims: Statistics (success rates, percentages) embedded in the dataset without citations or an evaluation report to back them up. Research references that are vague ("documented in 2025 research study") or missing entirely.
• Consistency: Do all samples have the required fields the code expects? Are there samples missing fields that other samples have (e.g., prompt vs attack_description when the code only reads prompt)?
• Realism: Do the scenarios describe situations that could plausibly occur? Are there scenarios that describe real-world mechanisms inaccurately (e.g., describing password storage or API access in ways that don't reflect how these systems actually work)?
• Data provenance chain: If the eval claims data comes from an external source (HuggingFace dataset, academic benchmark, GitHub repo), is there a code path that loads from that source? If samples are hardcoded inline but claimed to come from an external dataset, note this — the provenance is unverifiable from the code alone. This is distinct from data quality; it is about whether the stated origin can be confirmed.
• Source format consistency: If an external source is claimed, do the samples match what that source would produce? For example, a dataset claiming to be from a specific benchmark should match that benchmark's known structure, field names, and content style. Samples that don't match the claimed source's format may be fabricated or heavily modified.

These observations feed into Phase 2 (Claims Coherence) and Phase 4 (Dataset Validity) and should be recorded in NOTES.md.

Phase 2: Evaluate Overall Coherence

Before diving into detailed checks, step back and ask: does this evaluation make sense as a coherent whole? This phase catches fundamental problems — fabricated data, false claims, mechanisms that don't work as described — that the detailed checks in Phases 3-5 are not designed to detect.

The approach is straightforward: collect every verifiable claim the evaluation makes, then check each one against the actual code and data. An evaluation that makes false or unsupported claims about its data, mechanisms, or capabilities cannot produce trustworthy results, regardless of how well its mechanics work.

2.1 Collect Claims

Read through everything gathered in Phase 1 — README, eval.yaml, docstrings, code comments, dataset comments, any referenced papers — and extract every verifiable claim. Record each in NOTES.md with its source location. Things to look for:

• Data provenance: Any statement about where data comes from. "Samples from X dataset", "based on Y benchmark", "drawn from Z corpus", "N attacks from [source]".
• Mechanism properties: How a technique works or what properties it has. "Non-erasable", "permanent", "accumulates across steps", "cannot be fooled by X", "runs in microseconds".
• Capability claims: What the eval claims to test or detect. "Detects multi-step attacks", "measures vulnerability to X", "evaluates resilience against Y".
• Performance claims: Numbers, percentages, or comparative statements. "Achieves 95% F1", "catches 100% of scenarios", "outperforms baseline".
• Attribution: Claims that content, methodology, or ideas come from specific external sources or papers.

Do not skip claims that seem obviously true. A fabricated evaluation will contain a mix of true and false claims — the true-sounding ones provide false credibility and are the most important to verify.

2.2 Verify Claims

For each claim, check it against the code and data. The categories below are guidance on what to look for — not every claim will fit neatly into a category, and you should investigate anything that seems wrong.

Data provenance:

• Is there a code path that loads from the claimed source, or are the samples hardcoded/generated locally?
• If you can access the claimed source (HuggingFace dataset, GitHub repo, paper), do the samples match its format, field names, and content style?
• Are claimed dataset sizes consistent with the actual source?
• Do cited papers and repos exist and describe what the eval claims they describe?

Mechanism properties:

• Does the code actually implement what the documentation describes? Compare the README's explanation against the actual implementation.
• Are claimed properties logically or mathematically sound? You don't need to be a domain expert, but apply basic reasoning. If something is claimed to be "permanent" or "irreversible", ask whether there is any operation that could undo it. If something claims to "accumulate across steps", check whether the eval actually has multiple steps.
• Does each component contribute meaningfully? If the eval has multiple scoring or detection components, could any be removed without materially changing results? A component that never affects the outcome is dead weight — and may exist to make the eval appear more sophisticated than it is.

Capability and performance claims:

• If the eval claims to test capability X, does it actually contain test cases that exercise X? Be literal: "multi-step detection" requires multi-step scenarios; "social engineering detection" requires social engineering samples.
• Where do performance numbers come from? This eval's own runs, or a different implementation on different data? Are the results presented in a way that could mislead about their source?
• Could you reproduce the claimed results by running this eval?

2.3 Assess Overall Coherence

After checking individual claims, consider the evaluation as a whole:

• Do the claims, code, data, and scoring tell a coherent and consistent story?
• Are there signs that the eval was assembled without adequate understanding of its own mechanisms — e.g., documentation that describes capabilities the code doesn't implement, or test data that doesn't exercise the claimed functionality?
• Is the complexity of the implementation justified by the evaluation's purpose, or does it appear to exist for its own sake?

Trust your judgement here. If something feels off — overly complex mechanisms for a simple task, impressive-sounding math that doesn't connect to the scoring logic, data that seems too clean or generic — it is worth investigating further even if you can't immediately articulate why.

2.4 Record Findings

Create a claims inventory table in NOTES.md:

#	Claim	Source	Category	Verification Result	Severity
1	"samples from X dataset"	README line N	Data provenance	No loading code; inline samples only	High
2	"non-erasable scar"	README, docstring	Mechanism	Operation is trivially reversible	High

For each claim marked Medium or High, write a brief explanation of what the claim states, what the evidence shows, and why it matters for the eval's validity.

A single High-severity false claim — especially fabricated data provenance or a demonstrably false mechanism claim — should generally drive the overall validity rating to "Significant Validity Issues" regardless of how well the mechanics work. An evaluation built on false claims cannot produce trustworthy results.

Phase 3: Assess Name Validity

Determine whether the evaluation's name accurately represents what it measures. The checks below are common patterns to look for — use your judgement and investigate anything that seems misleading, even if it doesn't fit a listed pattern.

3.1 Identify the Claimed Scope

From the eval name, README, eval.yaml description, and group classification, determine what capability the evaluation claims to measure. Be specific — "reasoning" is a claim, "multi-step arithmetic reasoning" is a more precise claim.

3.2 Check for Scope Mismatch

Compare the claimed scope against what the samples actually test. Common problems:

• Overly broad name: The name implies a general capability, but the dataset only tests a narrow subset. Example: an eval named "Compassion" that only tests animal welfare in travel scenarios should be named more specifically.
• Overselling: The name implies a level of difficulty or danger that the samples don't support. Example: an eval named "Critical Infrastructure Hacking" that only contains basic CTF challenges.
• Measuring statements vs behaviour: The eval claims to measure a behaviour (e.g., "power seeking") but only asks the model questions about that behaviour without giving it the opportunity to actually exhibit it. This is a critical distinction — asking a model "would you seek power?" measures what the model says about power seeking, not whether it actually seeks power. For this to be a valid concern, the model must lack the affordances (tools, environment) to perform the behaviour. If the model has relevant tools and sandbox access, the eval may genuinely test the behaviour even if it starts with a question.
• Unsubstantiated claims in documentation: The README, eval.yaml, or dataset contain specific statistics, success rates, or research citations that are not backed by an evaluation report, paper, or verifiable source. Flag these separately from name validity -- they are credibility issues that affect how the evaluation's results will be interpreted.
• Accurate name: The name matches the actual scope of the evaluation. Note this as a positive finding.

3.3 Record Findings

For each issue found, note:

• The specific mismatch (what the name implies vs what the eval actually tests)
• Severity: High (actively misleading — someone reading only the name would have a materially wrong understanding), Medium (imprecise but not misleading — the name is too broad but not wrong), Low (minor quibble — could be more specific but unlikely to cause confusion)
• A suggested alternative name, if applicable

Phase 4: Assess Dataset Validity

Determine whether it is reasonably possible for a model to both succeed and fail at each sample given the available environment and affordances. The checks below are starting points — follow any thread that suggests success or failure might be impossible or trivial.

4.1 Identify Model Affordances

Based on Phase 1, list what the model can actually do:

• What tools does it have? (bash, python, web browse, custom tools)
• What sandbox resources are available? (servers, files, databases)
• What information is in the prompt?
• Can the model take actions, or only answer questions?

4.2 Check for Impossible Success

For each sample type (or a representative subset if the dataset is large), ask: Could a capable model plausibly succeed at this?

Red flags:

• The task references resources that don't exist in the sandbox (files, servers, APIs)
• The task requires information that isn't available in the prompt or discoverable through tools
• The task requires capabilities that aren't provided (e.g., sending an email without an email tool)
• The submission mechanism is unclear or doesn't match what's available (e.g., "submit your answer" but no submit tool, and the scorer doesn't check the conversation)

For agentic evaluations, attempt to trace the critical path from prompt to success:

1. What must the model do first?
2. What tools/actions are needed at each step?
3. Is each step achievable with the available affordances?

If you can identify a step that cannot be completed, flag it. You do not need to be able to solve the task yourself — models are broadly capable. But you do need to verify that the path to success exists (the right tools, files, and services are available).

4.3 Check for Impossible Failure

Ask: Could a model plausibly fail at this? This is less common but still important.

Red flags:

• The task asks the model to reveal specific information (credentials, system prompts, PII, configuration details, database contents), but that information does not exist in the model's environment (no sandbox files, no system message containing it, no tool to retrieve it). If the model "complies," it can only hallucinate — the scorer cannot distinguish genuine leaks from fabricated responses. For the failure to be meaningful, the information must be real and verifiable.
• The task asks the model to avoid doing something, but the model lacks the tools to do it (e.g., "don't send Bitcoin" when there's no Bitcoin tool — the model can only succeed)
• The scorer gives full marks regardless of the model's actual response
• The task is trivially easy with no meaningful way to get it wrong (e.g., exact text to copy is in the prompt and the scorer just checks for that text)

For safety/refusal evaluations specifically: Walk through each scenario type and ask: "If the model fully complied with this attack, would the compliance be verifiable?" If the scenario asks the model to reveal a password, does a real password exist somewhere the model could access? If the scenario asks the model to leak its system prompt, is there a system prompt set in the task that the scorer could compare against? Unverifiable compliance means the scorer is measuring whether the model appears to comply, not whether it actually complied — flag this as a validity concern.

4.4 Check Prompt Clarity

For agentic evaluations, check whether the prompt gives the model enough information to attempt the task:

• Is the objective clear?
• Is the expected output format specified (or does the scorer handle any format)?
• If the model needs to use specific tools or interact with specific services, is this indicated?

This is not about prompt engineering quality — it's about whether a capable model could understand what it needs to do.

4.5 Record Findings

For each issue found, note:

• The specific problem (what can't be done and why)
• Which samples are affected (specific IDs or "all samples" or "all samples of type X")
• Severity: High (success or failure is impossible for affected samples — the evaluation cannot produce valid signal for these), Medium (success or failure is technically possible but unreasonably difficult due to missing information or unclear instructions), Low (minor friction that a capable model could work around)
• A suggested fix, if apparent (e.g., "add a file submission tool", "start the server in compose.yaml", "clarify the submission format in the prompt")

Phase 5: Assess Scoring Validity

Determine whether the scorer measures actual task completion rather than proxies. As with other phases, the checks below are guidelines — investigate any aspect of the scoring that seems like it could produce misleading results.

5.1 Identify the Ground Truth

What does "success" actually mean for this evaluation? This should come from the task's purpose, not just what the scorer checks.

For evaluations where success means the model refuses a harmful request: also define what failure (compliance) looks like concretely, and ask whether the scorer can actually verify it. If the model claims to reveal a secret, does the scorer check that the secret is real? If not, the scorer is measuring the appearance of compliance rather than actual compliance — this is a form of weak proxy that should be flagged in 5.2.

5.2 Check Scorer-to-Ground-Truth Alignment

Compare what the scorer checks against the ground truth:

• Direct measurement (best): The scorer checks the actual outcome. Examples: running submitted code on a test set, checking that a password was extracted from a server, verifying a file was correctly modified. Note this as a positive finding.
• Strong proxy: The scorer checks something that is very closely tied to the outcome but not identical. Example: checking that the correct answer string is present in the response for a Q&A task with unambiguous answers. This is generally acceptable for Q&A evaluations.
• Weak proxy (problematic): The scorer checks something loosely correlated with success. Examples: checking that code compiles without running it, checking that hyperparameters were set correctly without running the model, checking that a port was accessed without verifying the exploit worked.
• Substring matching for natural language (often problematic): If the scorer uses substring matching to determine compliance, refusal, or other natural language properties, this is usually a weak proxy. LLM judges are preferred for natural language assessment. The exception is when the substring is itself the ground truth (e.g., a CTF flag, a password, a specific computed value that can only be obtained by completing the task).

5.3 Check for Scoring Edge Cases

• Could a model get a perfect score without actually completing the task? (e.g., by guessing the flag format, by triggering a scoring bug)
• Could a model complete the task correctly but receive a 0 score? (e.g., correct answer in wrong format, correct action but scorer checks wrong thing)
• For LLM-judged scorers: is the judge prompt specific enough to distinguish genuine completion from superficial compliance?

5.4 Record Findings

For each issue found, note:

• What the scorer checks vs what it should check
• Severity: High (scorer fundamentally doesn't measure the right thing — results cannot be trusted), Medium (scorer is a weak proxy — results are directionally useful but noisy), Low (minor edge case that affects a small fraction of samples)
• A suggested improvement, if apparent (e.g., "run the code on a test set instead of checking compilation", "use an LLM judge instead of substring matching", "check the file contents instead of checking the command was run")

Phase 6: Write Report

Create VALIDITY_REPORT.md in the output directory with the following structure:

# Evaluation Validity Report: <eval_name>

**Version**: <task version>
**Date**: <today's date>
**Group**: <eval.yaml group>

## Summary

<1-3 sentence overview: is this evaluation valid? What are the main concerns, if any?>

## Overall Validity Rating

<One of: **Valid**, **Valid with Minor Issues**, **Validity Concerns**, **Significant Validity Issues**>

<Brief justification for the rating>

## Claims Verification

### Claims Inventory
<Summary count: N claims checked, X verified, Y unverifiable, Z false/contradicted>

### Findings
<List of High and Medium severity claim issues, or "No issues found">

## Name Validity

### Claimed Scope
<What the name and documentation claim the eval measures>

### Actual Scope
<What the eval actually measures based on code analysis>

### Findings
<List of issues, or "No issues found">

## Dataset Validity

### Model Affordances
<Summary of what the model can do: tools, sandbox, information available>

### Findings
<List of issues, or "No issues found">

## Scoring Validity

### Scoring Approach
<How the scorer works>

### Ground Truth Alignment
<How well the scorer measures actual task completion>

### Findings
<List of issues, or "No issues found">

## Recommendations

<Prioritised list of suggested changes, if any. Each should include:
- What to change
- Why it matters
- Expected difficulty (trivial / moderate / significant)>

Phase 7: Wrap Up

1. Review NOTES.md and ensure all important observations made it into the report.
2. Tell the user the task is complete and where to find the report.
3. Highlight the overall validity rating and any High severity findings.