eval-quality-workflow

name

eval-quality-workflow

description

Fix or review a single evaluation against all EVALUATION_CHECKLIST.md standards. Use "fix" mode to refactor an eval into compliance, or "review" mode to assess compliance without making changes. Use when user asks to fix, review, or check an evaluation's quality. Trigger when the user asks you to run the "Fix An Evaluation" or "Review An Evaluation" workflow. Do NOT use for reviewing ALL evals against a single code quality standard (use code-quality-review-all instead).

Evaluation Quality — Fix or Review

This skill covers two closely related workflows for a single evaluation in src/inspect_evals/:

• Fix An Evaluation: Refactor the evaluation to comply with EVALUATION_CHECKLIST.md
• Review An Evaluation: Assess compliance without making changes

Identifying the Evaluation

If the user has given you a name, that takes priority. If you were just building an evaluation, or the user has uncommitted code for one specific evaluation, you can assume that's the correct one. If you are not confident which evaluation to look at, ask the user.

Fix An Evaluation

Our standards are in EVALUATION_CHECKLIST.md, with links to BEST_PRACTICES.md and CONTRIBUTING.md. Your job is to refactor the evaluation to meet these standards.

1. Set up the working directory:
   1. If the user provides specific instructions about any step, assume the user's instructions override these instructions.
   2. If there is no evaluation name, ask the user for one.
   3. The evaluation name should be the eval folder name plus its version (from the @task function's version argument). For instance, GPQA version 1.1.2 becomes "gpqa_1_1_2". If this exact folder name already exists, add a number to it via "gpqa_1_1_2_analysis2". This name will be referred to as <eval_name>.
   4. Create a folder called agent_artefacts/<eval_name>/fix if it isn't present.
   5. Whenever you create a .md file as part of this workflow, assume it is made in .