eval-report-workflow

name	eval-report-workflow
description	Create an evaluation report for a README by selecting models, estimating costs, running evaluations, and formatting results tables. Use when user asks to make/create/generate an evaluation report. Trigger when the user asks you to run the "Make An Evaluation Report" workflow.

Make an Evaluation Report

This workflow drives tools/evaluation_report.py, which reads a per-eval report_config.yaml and produces a full reproducible report.md (results table, reference comparison, per-category breakdowns, token totals, approximate cost) plus header-only JSON copies of the input logs under results/. The report_config.yaml, regenerated report.md, and results/ folder are committed alongside the eval's eval.yaml.

Report Formatting

The evaluation report included in the README.md is the rendered report.md produced by tools/evaluation_report.py. It should run on the entire dataset or $5 of compute per model, whichever is cheaper. Use the token count from smaller runs to make this prediction.

A typical rendered report looks like this:

# Evaluation Report

## Implementation Details

Brief description of any deviations from the paper, known limitations, etc.

## Results

| Model         | Inspect (accuracy) | Reference | Δ      | Samples | Stderr | Time |
| ------------- | ------------------ | --------- | ------ | ------- | ------ | ---- |
| openai/...    | 0.600              | 0.580     | +0.020 | 100/100 | 0.049  | 18s  |
| anthropic/... | 0.400              | 0.420     | -0.020 | 100/100 | 0.049  | 6s   |

_Reference: Paper, Table 3_

## Reproducibility Information

- Samples: 100 / 100 per model
- Run dates: 2026-04-29
- Versions: inspect_ai=0.3.x, inspect_evals=0.x
- Models: ...
- Total tokens: 1,234,567
- Approximate cost: $0.42 USD (prices as of 2026-04)

Reproduction commands: ...

Register entries: for register entries (register/<name>/eval.yaml), populate the optional evaluation_report block in eval.yaml instead of editing README.md directly — the README is regenerated from the YAML by make check. The block accepts timestamp, a results list (with model, accuracy, and optionally provider, stderr, time, date), and notes. Extra fields at either level are allowed for eval-specific metric columns. See register/README.md for the schema.

If the eval.yaml file includes an arXiv paper, check that paper for the models used and human baselines. Include the human baseline in the notes section if it is present. If you can, select three models that would be suitable to check if this evaluation successfully replicates the original paper, including at least two different model providers.

If you cannot do this, or if there aren't three suitable candidates, use frontier models to fill out the remainder of the list. Currently, these are as follows:

Frontier Models

See references/frontier-models.md for the current list of frontier models and their costs. This file should be updated when models or prices change.

Workflow Steps

1. Set up the working directory:

   1. If the user provides specific instructions about any step, assume the user's instructions override these instructions.
   2. If there is no evaluation name, ask the user for one.
   3. The evaluation name should be the eval folder name plus its version (from the @task function's version argument). For instance, GPQA version 1.1.2 becomes "gpqa_1_1_2". If this exact folder name already exists, add a number to it via "gpqa_1_1_2_analysis2". This name will be referred to as <eval_name>.
   4. Create a folder called agent_artefacts/<eval_name>/evalreport if it isn't present.
   5. Whenever you create a .md file as part of this workflow, assume it is made in agent_artefacts/<eval_name>/evalreport.
   6. Copy EVALUATION_CHECKLIST.md to the folder.
   7. Create a NOTES.md file for miscellaneous helpful notes. Err on the side of taking lots of notes. Create an UNCERTAINTIES.md file to note any uncertainties.

2. Read the Evaluation Report Guidelines.

3. Check to see if the README for the evaluation already has an evaluation report. If so, double-check with the user that they want it overwritten.

4. Read the main file in EVAL_NAME, which should be src/inspect_evals/EVAL_NAME/EVAL_NAME.py in order to see how many tasks there are.

5. Perform an initial test with 'uv run inspect eval inspect_evals/EVAL_NAME --model gpt-5.1-2025-11-13 --limit 5' to get estimated token counts. Use -T shuffle=True if possible to produce random samples - to see if it's possible, you'll need to check the evaluation itself.

6. Perform model selection as above to decide which models to run.

7. Select a method of randomisation of the samples that ensures all meaningfully different subsets of the data are checked. This is as simple as ensuring the dataset is shuffled in most cases. Explicitly track how many meaningfully different subsets exist.

8. Tell the user what commands to run and how much compute it is expected to cost.

   Base your compute calculation on the most expensive model in your list. Each model should be run on the same dataset size regardless of cost, hence we limit it via the most expensive one. If the task will be more expensive than $5 per model to run the full dataset, give the user your estimate for the cost of the full dataset. This means if there are multiple tasks to run, you'll need to split the cost among them according to token usage in the initial estimate. You should assume Gemini reasoning models take roughly 10x the tokens of the other models when performing these calculations.