メニュー
Create custom inspect-ai evaluation tasks through interacted, guided workflow.
Codex または Claude でインストール この Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。
SOC 職業分類に基づく
| name | create-inspect-task |
| description | Create custom inspect-ai evaluation tasks through interacted, guided workflow. |
You help users create custom inspect-ai evaluation tasks through an interactive, guided workflow. Create well-documented, reusable evaluation scripts that follow inspect-ai best practices.
Guide the user through designing and implementing a custom inspect-ai evaluation task. Create a complete, runnable task file and comprehensive documentation that explains the design decisions and usage.
This skill supports two modes:
When an experiment_summary.yaml file exists (created by design-experiment skill), extract configuration to pre-populate:
Usage: Run skill from experiment directory or provide path to experiment_summary.yaml
Create evaluation tasks from scratch without experiment context. User provides all configuration manually.
Usage: Run skill when no experiment exists or when creating general-purpose evaluation tasks
experiment_summary.yaml in current directorylogs/create-inspect-task.loglogs/create-inspect-task.logWhen operating in experiment-guided mode, extract the following information from the YAML structure:
experiment:
name: string
project: string
question: string
data:
training:
path: string
dataset_label: string
format: string
splits:
train: int
validation: int
test: int
models:
base:
- name: string
path: string
evaluation:
system_prompt: string
temperature: float
runs:
- name: string
type: string # "fine-tuned" or "control"
model: string
import yaml
from pathlib import Path
def extract_from_experiment_summary(path):
"""Extract configuration from experiment_summary.yaml"""
with open(path, 'r') as f:
config = yaml.safe_load(f)
# Extract dataset configuration
dataset_path = config['data']['training']['path']
dataset_format = config['data']['training']['format']
dataset_splits = config['data']['training']['splits']
# Extract system prompt from evaluation section
system_prompt = config['evaluation']['system_prompt']
# Extract research question
research_question = config['experiment']['question']
project = config['experiment']['project']
# Extract model information (first base model)
base_models = config['models']['base']
model_name = base_models[0]['name'] if base_models else None
model_path = base_models[0]['path'] if base_models else None
# Extract run names for documentation examples
run_names = [run['name'] for run in config['runs']]
control_runs = [run['name'] for run in config['runs'] if run['type'] == 'control']
return {
'dataset_path': dataset_path,
'dataset_format': dataset_format,
'dataset_splits': dataset_splits,
'system_prompt': system_prompt,
'research_question': research_question,
'project': project,
'model_name': model_name,
'model_path': model_path,
'run_names': run_names,
'control_runs': control_runs
}
From experiment section:
question → Research question/objective (informs evaluation goal)project → Blueprint directory name under blueprints/ (pins the task family)From data.training section:
path → Dataset path for evaluationformat → Dataset format (json)splits → Sample counts (use test split for evaluation)From models.base[] section:
name → Model identifierpath → Full path to base model (for usage examples)From evaluation section:
system_prompt → from controls.system_prompt (the single source, propagated to eval)temperature → Default temperature settingFrom runs[] section:
name → Run identifiers (for documentation)type → Filter for "control" runs that need evaluationAfter extraction, show the user what was found:
## Configuration Extracted from Experiment
I found the following configuration in your experiment:
**Dataset:**
- Path: `{ck_data_dir}/capitalization/words_4L_80P_300.json`
- Format: JSON
- Splits: train (240), test (60)
**Models:**
- Llama-3.2-1B-Instruct
- Path: `/scratch/gpfs/.../pretrained-llms/Llama-3.2-1B-Instruct`
**System Prompt:**
{extracted_prompt or "(none)"}
**Research Question:**
{extracted_question}
I'll use this information to help configure your evaluation task. You can override any of these settings if needed.
Check extracted information:
ls)ls)If validation fails:
IMPORTANT: Create a detailed log file at {task_dir}/logs/create-inspect-task.log that records all questions, answers, and decisions made during task creation.
[YYYY-MM-DD HH:MM:SS] ACTION: Description
Details: {specifics}
Result: {outcome}
[2025-10-24 14:30:00] MODE_SELECTION: Experiment-guided mode
Details: Found experiment_summary.yaml at /scratch/gpfs/MSALGANIK/mjs3/cap_4L_lora_lr_sweep/experiment_summary.yaml
Result: User confirmed to use experiment configuration
[2025-10-24 14:30:05] EXTRACT_CONFIG: Reading experiment_summary.yaml
Details: Parsing YAML structure: experiment, data, models, evaluation sections
Result: Successfully extracted configuration
[2025-10-24 14:30:10] EXTRACTED_DATASET: Dataset configuration
Details: Path: {ck_data_dir}/capitalization/words_4L_80P_300.json
Format: JSON, Splits: train (240), test (60)
Result: Verified dataset exists (43KB)
[2025-10-24 14:30:15] EXTRACTED_SYSTEM_PROMPT: System prompt from experiment
Details: Prompt: "" (empty - no system message)
Result: Will use the system prompt from `controls.system_prompt` (here: empty)
[2025-10-24 14:30:20] EXTRACTED_RESEARCH_QUESTION: Scientific objective
Details: Compare LoRA ranks and learning rates for capitalization task
Result: Will design evaluation to measure exact match accuracy
[2025-10-24 14:30:25] EVALUATION_OBJECTIVE: User wants to evaluate capitalization accuracy
Details: Exact match (case-sensitive), using experiment dataset
Result: Will use match(location="exact", ignore_case=False) scorer for strict evaluation
[2025-10-24 14:30:30] SOLVER_CONFIG: Designing solver chain
Details: system_message(""), prompt_template("{prompt}"), generate(temp=0.0)
Result: Matches training configuration for consistency
[2025-10-24 14:30:00] MODE_SELECTION: Standalone mode
Details: No experiment_summary.yaml found
Result: User will provide all configuration manually
[2025-10-24 14:30:05] EVALUATION_OBJECTIVE: User wants to evaluate sentiment classification
Details: Binary classification (positive/negative), using custom dataset in JSON format
Result: Will use match() scorer for exact matching, temperature=0.0 for consistency
[2025-10-24 14:30:15] DATASET_CONFIG: Selected JSON dataset format
Details: Dataset path: /scratch/gpfs/MSALGANIK/niznik/data/sentiment_test.json
Field mapping: input="text", target="sentiment"
Result: Will use hf_dataset with json format and custom record_to_sample function
What do you want to evaluate?
What defines a correct answer?
What dataset format do you have?
.json or .jsonl)Where is the dataset located?
What are the field names?
Dataset structure specifics:
Example questions:
{'train': [...], 'test': [...]}?"System message:
Prompt template:
"{prompt}" (direct input)Generation parameters:
Common solver patterns:
[system_message(""), prompt_template("{prompt}"), generate()][chain_of_thought(), generate()][multiple_choice()] (don't add separate generate())[prompt_template("Answer: {prompt}\n"), generate()]Based on evaluation objective, suggest scorers:
cruijff_kit custom scorers (preferred for experiment tasks — driven by the scorers: block in eval.yaml):
Rather than hard-coding a scorer in the task, read the scorers: list from config_path and build it via the shared registry. This lets the experiment design (not the task code) pick the scorer:
from cruijff_kit.tools.inspect.scorers import (
build_scorers,
configured_scorers_require_logprobs,
)
scorers = build_scorers(config) # falls back to DEFAULT_SCORERS if no scorers: block
The scorers: block is a list of {name, params} entries:
scorers:
- name: match
- name: risk_scorer
params:
option_tokens: ["0", "1"]
Registry names: match, includes, risk_scorer, numeric_risk_scorer, continuous_scorer.
logprobs contract: logprob-based scorers (e.g. risk_scorer) set requires_logprobs = True on their factory. Call configured_scorers_require_logprobs(config) and auto-enable logprob capture when it returns True:
scorer_needs_logprobs = configured_scorers_require_logprobs(config)
enable_logprobs = bool(logprobs) or (logprobs is None and scorer_needs_logprobs)
if enable_logprobs:
generate_config = GenerateConfig(logprobs=True, top_logprobs=top_logprobs)
else:
generate_config = GenerateConfig()
# ... and fail loudly if the user passed logprobs=False while configuring a
# scorer that needs them, rather than silently producing unscored logs.
Note: risk_scorer accuracy is exact-string-match (completion == target), not argmax — it conflates output format with judgment. Keep that in mind when interpreting its accuracy.
For exact matching (inspect-ai built-ins — fine for standalone tasks without the registry):
match() - Target appears at beginning/end; ignores case, whitespace, punctuation
location="begin"/"end"/"any", ignore_case=True/Falseexact() - Precise matching after normalizationincludes() - Target appears anywhere in output
ignore_case=True/FalseFor multiple choice:
choice() - Works with multiple_choice() solverFor pattern extraction:
pattern() - Extract answer using regex
For model-graded evaluation:
model_graded_qa() - Another model assesses answer quality
partial_credit=True/False, custom templatemodel_graded_fact() - Checks if specific facts appearFor numeric/F1 scoring:
f1() - F1 score for text overlapMultiple scorers:
[match(), includes()] to get multiple scoresShould the task accept parameters for flexibility?
Standard parameters — match the current blueprint tasks so the task drops into the eval.yaml pipeline (cleanest reference: blueprints/capitalization/inspect_task.py):
data_path — Path to the dataset JSON (required; no default).config_path — Path to eval.yaml. The task reads prompt, system_prompt, and (optionally) the scorers: block from it. setup_inspect.py auto-derives this from the config file's own location — you do not pass it by hand.vis_label — Optional label appended to the task name (f"{name}_{vis_label}" if vis_label else name). Required for the viz pipeline: viz_helpers reads vis_label from the eval log's task_args to dedup and label runs. Omit it and multi-variant runs collide in the visualizations.split — Which split to evaluate (e.g. "test", "validation").use_chat_template — True for instruct models (adds a system_message solver), False for base models (plain text completion).temperature / max_tokens — Generation params. Greedy default is temperature=1e-7.logprobs / top_logprobs — Capture top logprobs. Leave logprobs defaulting to None (auto) so it enables only when a configured scorer needs them — see Scorer Selection.assistant_prefix — Optional text to seed the assistant turn.These names are not arbitrary. setup_inspect.py maps a fixed TASK_ARG_KEYS set onto -T flags — commonly the likes of:
data_path, config_path, vis_label, split, temperature, max_tokens, …
TASK_ARG_KEYS in setup_inspect.py is the source of truth for the full set — read it there rather than trusting this list to stay complete. A parameter outside that set will not receive a value from eval.yaml (you'd get a startup warning from load_eval_config about an unconsumed key); add genuinely new task args to TASK_ARG_KEYS if the task needs them.
Benefits of parameters:
How to pass parameters:
inspect eval task.py -T param_name=value
How will the model be specified?
Option 1: CLI specification (most flexible)
inspect eval task.py --model hf/local -M model_path=/path/to/modelOption 2: Experiment pipeline (eval.yaml)
setup_inspect.py from eval.yamlprompt/system_prompt/scorers from config_path — it never resolves the modelscaffold-inspect runs tasks; see Generated Task Pattern belowOption 3: Hard-coded in task
Create two files:
{task_name}_task.pyThe complete, runnable inspect-ai task following best practices.
File naming convention:
sentiment_classification_task.pymath_reasoning_task.py{domain}_{type}_task.pyRequired components:
from inspect_ai import Task, task
from inspect_ai.dataset import json_dataset, hf_dataset, FieldSpec
from inspect_ai.solver import chain, generate, prompt_template, system_message
from inspect_ai.scorer import match, includes
@task
def my_task(param1: str = "default"):
"""
Brief description of what this task evaluates.
Args:
param1: Description of parameter
Returns:
Task: Configured inspect-ai task
"""
# Dataset loading
dataset = ...
# Solver chain
solver = chain(
system_message("..."),
prompt_template("{prompt}"),
generate(temperature=0.0)
)
# Return task
return Task(
dataset=dataset,
solver=solver,
scorer=...
)
Best practices to follow:
{task_name}_design.mdComprehensive documentation of design decisions.
Required sections:
# {Task Name} Evaluation Task
**Created:** {timestamp}
**Inspect-AI Version:** {version if known}
## Evaluation Objective
{What this task evaluates and why}
## Dataset Configuration
**Format:** {JSON/HuggingFace/etc.}
**Location:** `{full_path_to_dataset}`
**Size:** {number of samples if known}
**Field Mapping:**
- Input field: `{field_name}`
- Target field: `{field_name}`
- Metadata fields: `{field_names or "none"}`
**Loading Method:**
{Description of how dataset is loaded}
**Data Structure:**
{Explanation of JSON structure, splits, etc.}
## Solver Chain
**Components:**
1. {Solver 1}: {Purpose}
2. {Solver 2}: {Purpose}
3. ...
**System Message:**
{system message text or "none"}
**Prompt Template:**
{template or "direct input"}
**Generation Parameters:**
- Temperature: {value} - {rationale}
- Max tokens: {value or "default"} - {rationale}
- {Other parameters if any}
**Rationale:**
{Why this solver chain was chosen}
## Scorer Configuration
**Primary Scorer:** `{scorer_name}()`
**Options:**
- {option1}: {value} - {reason}
- {option2}: {value} - {reason}
**Additional Scorers:**
{List if multiple scorers used, or "none"}
**Rationale:**
{Why this scorer is appropriate for the task}
## Task Parameters
| Parameter | Type | Default | Purpose |
|-----------|------|---------|---------|
| {param1} | {type} | {default} | {description} |
**Parameter Usage:**
```bash
inspect eval {task_file}.py -T {param}={value}
Recommended usage:
inspect eval {task_file}.py --model hf/local -M model_path=/path/to/model
{Any specific notes about model compatibility}
Basic evaluation:
inspect eval {task_name}_task.py --model hf/local -M model_path=/path/to/model
With parameters:
inspect eval {task_name}_task.py --model hf/local -M model_path=/path/to/model -T temperature=0.5
Evaluating fine-tuned model: {if applicable}
inspect eval {task_name}_task.py@{task_name} --model hf/local \
-M model_path=/path/to/run/artifacts/epoch_0 \
-T data_path=/path/to/dataset.json \
-T config_path=/path/to/eval.yaml
Inspect-ai will create:
logs/{task_name}_{timestamp}.eval - Evaluation results log{If known, describe expected baseline performance or what good performance looks like}
{Any additional considerations, limitations, or future improvements}
## Code Generation Guidelines
### Dataset Loading Patterns
**JSON with nested splits:**
```python
from inspect_ai.dataset import hf_dataset
def record_to_sample(record):
return Sample(
input=record["input"],
target=record["output"]
)
dataset = hf_dataset(
path="json",
data_files="/path/to/data.json",
field="test", # Access the "test" split
split="train", # Don't get confused - this refers to top-level split
sample_fields=record_to_sample
)
JSONL (one JSON object per line):
from inspect_ai.dataset import json_dataset
def record_to_sample(record):
return Sample(
input=record["question"],
target=record["answer"]
)
dataset = json_dataset(
"/path/to/data.jsonl",
record_to_sample
)
HuggingFace dataset:
from inspect_ai.dataset import hf_dataset, FieldSpec
dataset = hf_dataset(
path="username/dataset-name",
split="test",
sample_fields=FieldSpec(
input="question",
target="answer",
metadata=["category", "difficulty"] # Preserve metadata
)
)
Simple generation:
from inspect_ai.solver import chain, generate, prompt_template, system_message
solver = chain(
system_message(""), # Empty if no system message needed
prompt_template("{prompt}"), # Direct input
generate(temperature=0.0)
)
With system message and custom template:
solver = chain(
system_message("You are an expert classifier. Respond with only the category label."),
prompt_template("Text: {prompt}\n\nCategory:"),
generate(temperature=0.0, max_tokens=50)
)
Chain-of-thought:
from inspect_ai.solver import chain_of_thought, generate
solver = chain(
chain_of_thought(), # Adds "Let's think step by step" prompt
generate(temperature=0.0)
)
Multiple choice:
from inspect_ai.solver import multiple_choice
solver = multiple_choice() # Don't add generate() separately
# Or with chain-of-thought:
solver = multiple_choice(cot=True)
Exact matching (case-insensitive):
from inspect_ai.scorer import match
scorer = match() # Default: ignore case, whitespace, punctuation
# Or customize:
scorer = match(location="exact", ignore_case=False)
Substring matching:
from inspect_ai.scorer import includes
scorer = includes() # Default: case-sensitive
# Or:
scorer = includes(ignore_case=True)
Multiple scorers:
scorer = [
match("exact", ignore_case=False),
includes(ignore_case=False)
]
# Results will show scores from both
Model-graded:
from inspect_ai.scorer import model_graded_qa
scorer = model_graded_qa(
partial_credit=True, # Allow 0.5 scores
model="openai/gpt-4o" # Specify grading model
)
When creating tasks for an experiment, the task does not read fine-tuning configs directly. Instead it reads its prompt/scorers config from the eval.yaml that scaffold-inspect writes, via the auto-derived config_path:
design-experiment produces experiment_summary.yaml (research question, data, models, controls.system_prompt, the scorers: block).scaffold-inspect writes one eval.yaml per (run, task, epoch) cell, carrying prompt, system_prompt, scorers, data_path, vis_label, etc.setup_inspect.py renders the SLURM script: it auto-derives config_path (the path to that eval.yaml), maps TASK_ARG_KEYS onto -T flags, and the task reads prompt/system_prompt/scorers back out of config_path at runtime.So a generated task's job is: accept the standard params, read prompt/system_prompt (and optionally the scorers: block) from config_path, build the dataset/solver/scorer, and name itself with vis_label.
For tasks integrated with experiments (mirrors blueprints/capitalization/inspect_task.py, the cleanest reference):
import yaml
from inspect_ai import Task, task
from inspect_ai.dataset import hf_dataset, Sample
from inspect_ai.model import GenerateConfig
from inspect_ai.solver import chain, generate, system_message
from cruijff_kit.tools.inspect.scorers import (
build_scorers,
configured_scorers_require_logprobs,
)
@task
def my_task(
data_path: str,
config_path: str = "",
split: str = "test",
temperature: float = 1e-7,
max_tokens: int = 20,
use_chat_template: bool = True,
logprobs: bool | None = None,
top_logprobs: int = 20,
vis_label: str = "",
) -> Task:
"""
Evaluate a model on <task description>.
Args:
data_path: Path to the dataset JSON.
config_path: Path to eval.yaml (reads prompt/system_prompt/scorers).
split: Which split to evaluate (default: "test").
temperature: Generation temperature (greedy default 1e-7).
max_tokens: Max tokens to generate.
use_chat_template: True for instruct models (adds system_message solver),
False for base models (plain text completion).
logprobs: None = auto (enable iff a configured scorer needs them).
top_logprobs: Top-k logprobs to capture when enabled.
vis_label: Optional suffix for the task name; read by viz_helpers for
dedup/labeling across a multi-variant experiment.
Returns:
Task: Configured inspect-ai task
"""
# Task name carries vis_label so the viz pipeline can label/dedup runs.
task_name = f"my_task_{vis_label}" if vis_label else "my_task"
prompt_str = "{input}"
system_prompt = ""
config: dict = {}
if config_path:
with open(config_path, "r") as f:
config = yaml.safe_load(f) or {}
prompt_str = config.get("prompt", "{input}")
system_prompt = config.get("system_prompt", "")
def record_to_sample(record):
return Sample(
input=prompt_str.format(input=record["input"]),
target=record["output"],
metadata=record.get("metadata", {}),
)
dataset = hf_dataset(
path="json",
data_files=data_path,
field=split,
split="train", # HuggingFace quirk — always "train" here
sample_fields=record_to_sample,
)
if use_chat_template:
solver = chain(
system_message(system_prompt),
generate(temperature=temperature, max_tokens=max_tokens),
)
else:
solver = chain(generate(temperature=temperature, max_tokens=max_tokens))
# Scorer + logprobs driven by the eval config's scorers: block.
scorers = build_scorers(config)
scorer_needs_logprobs = configured_scorers_require_logprobs(config)
if logprobs is False and scorer_needs_logprobs:
raise ValueError(
"A configured scorer requires logprobs, but logprobs=False was set. "
"Drop the override or remove the logprob-dependent scorer."
)
enable_logprobs = bool(logprobs) or (logprobs is None and scorer_needs_logprobs)
generate_config = (
GenerateConfig(logprobs=True, top_logprobs=top_logprobs)
if enable_logprobs
else GenerateConfig()
)
return Task(
name=task_name,
dataset=dataset,
solver=solver,
scorer=scorers,
config=generate_config,
)
For a standalone task with no experiment context, drop the config_path/build_scorers/logprobs machinery and hard-code a built-in scorer (e.g. scorer=match(...)) — but keep vis_label (and pass it yourself via -T vis_label=…, since there's no eval.yaml to auto-map it) if the eval logs will feed the viz pipeline:
from inspect_ai import Task, task
from inspect_ai.dataset import hf_dataset, Sample
from inspect_ai.scorer import match
from inspect_ai.solver import generate
@task
def my_task(
data_path: str,
split: str = "test",
temperature: float = 1e-7,
max_tokens: int = 20,
vis_label: str = "",
) -> Task:
"""Standalone eval on <task description> (no eval.yaml)."""
task_name = f"my_task_{vis_label}" if vis_label else "my_task"
def record_to_sample(record):
return Sample(
input=record["input"],
target=record["output"],
metadata=record.get("metadata", {}),
)
dataset = hf_dataset(
path="json",
data_files=data_path,
field=split,
split="train", # HuggingFace quirk — always "train" here
sample_fields=record_to_sample,
)
return Task(
name=task_name,
dataset=dataset,
solver=generate(temperature=temperature, max_tokens=max_tokens),
scorer=match(), # hard-coded — no scorers: block to read
)
In the experiment pipeline you don't invoke inspect eval by hand — scaffold-inspect + setup_inspect.py generate the SLURM cell. For a quick manual smoke test of a generated task:
inspect eval my_task.py@my_task \
--model hf/local \
-M model_path=/scratch/gpfs/MSALGANIK/pretrained-llms/Llama-3.2-1B-Instruct \
-T data_path=/path/to/dataset.json \
-T config_path=/path/to/eval.yaml \
-T vis_label=smoke \
--limit 5
This task pattern integrates with setup_inspect.py, which renders eval SLURM scripts from a template. The scaffold-inspect agent writes eval.yaml (referencing the task script created here) and calls:
python src/tools/inspect/setup_inspect.py \
--config eval.yaml \
--model_name Llama-3.2-1B-Instruct
Before finishing, verify:
@taskAdditional checks for experiment-guided mode:
controls.system_prompt (single source)data_path + config_path and reads prompt/system_prompt/scorers from config_pathvis_label and folds it into the task namesetup_inspect.py's TASK_ARG_KEYSAfter creating the task, guide user:
Test the task:
# Validate syntax
python -m py_compile {task_file}.py
# Test with small sample
inspect eval {task_file}.py --model {model} --limit 5
Run full evaluation:
inspect eval {task_file}.py --model {model}
View results:
inspect view
# Opens web UI to browse evaluation logs
Iterate if needed:
inspect score to re-score without re-running--limit 5)controls.system_prompt, propagated to eval (no separate eval copy to match)use_chat_template toggles instruct vs. base)config_path (the eval.yaml), and fold vis_label into the task nameIf dataset file not found:
If unsure about dataset format:
If scorer choice unclear:
Plan LLM fine-tuning and evaluation experiments. Use when the user wants to design a new experiment, plan training runs, or create an experiment_summary.yaml file.
Open-ended, Claude-driven exploration of a completed experiment — you decide per-experiment which figures matter and what the results mean. Optional and runnable any time after run-experiment (summarize-experiment is the required post-run step). Produces "Claude's Exploration": a non-deterministic report of what you examined and concluded, with an audit log.
Execute the complete experimental workflow - model optimization followed by evaluation - for all runs in a scaffolded experiment. Use after scaffold-experiment to submit jobs to SLURM.
Archive a completed experiment, preserving all experiment files while deleting bulk checkpoint artifacts. Use after summarize-experiment or explore-experiment when an experiment is complete and results have been reviewed.
Set up complete experimental infrastructure for all runs in a designed experiment. Orchestrates parallel generation of fine-tuning configs (via scaffold-torchtune) and evaluation configs (via scaffold-inspect). Use after design-experiment to prepare configs before running experiments.
Create the standard summary of experiment results from a completed (fine-tuned and evaluated) experiment. Run this right after run-experiment as the required post-run step — captures key metrics (loss, accuracy, regression metrics) into summary.md.