| ukgovernmentbeis-inspect-ai-overview | Covers what Inspect is (an open-source Python framework for LLM evals by the UK AI Security Institute), how to install it, the canonical hello-world Task example, and the full sub-package surface area — read this first when orienting to the repo or helping users get started. | |
| ukgovernmentbeis-inspect-ai-tasks | Covers the Task class and @task decorator as Inspect's fundamental evaluation unit — bundling dataset, solver, scorer, and all optional configuration (epochs, sandbox, approval, limits, checkpointing, early stopping). Read this when constructing or parameterizing tasks, applying task_with() overrides, understanding the four-layer configuration precedence, exposing solver parameters, or packaging tasks for distribution. | |
| ukgovernmentbeis-inspect-ai-datasets | Covers the full dataset contract in Inspect: the Sample Pydantic model and every field it carries (input, target, choices, id, metadata, sandbox, files, setup, checkpoint), the four built-in loaders (CSV, JSON/JSONL, Hugging Face, example), FieldSpec/record_to_sample field mapping, MemoryDataset for custom readers, filtering/shuffling/slicing knobs, and sandbox file-seeding patterns. | |
| ukgovernmentbeis-inspect-ai-solvers | Covers the Solver protocol and Generate callable that every Inspect evaluation is built on, the complete set of built-in solver components (generate, system_message, prompt_template, chain_of_thought, self_critique, multiple_choice, use_tools), the chain() compositor and @solver decorator for composite plans, and the TaskState data structure that every solver mutates. Includes concrete patterns for custom solvers, intermediate scoring, and the decision boundary between plain solvers and the Agent interface. | |
| ukgovernmentbeis-inspect-ai-scorers | Comprehensive reference for Inspect AI's built-in and model-graded scorers, the Score/Value/Metric type hierarchy, epoch reducers (including pass_at and pass_k), custom @scorer and @metric decorators, multi-value and multi-scorer patterns, and the inspect score CLI workflow. Covers exact/heuristic matchers, statistical metrics (accuracy, stderr, grouped), and sandbox-aware scoring. | |
| ukgovernmentbeis-inspect-ai-models-and-providers | Uniform model API (get_model, Model, ModelAPI) that normalizes 20+ inference backends behind one interface, plus the full GenerateConfig knob surface, the provider catalog with per-provider quirks and model args, adaptive-connections concurrency, batch mode, and reasoning-model effort/history/summary controls. | |
| ukgovernmentbeis-inspect-ai-tools | Covers every layer of Inspect's tool system: the @tool decorator and factory pattern, standard built-in tools (bash, python, web_search, text_editor, computer, web_browser, agentic helpers, and intervention tools ask_user/notify_user), MCP server integration via stdio/HTTP/sandbox transports, custom tool authoring (signatures, ToolError, ToolResult, ToolDef, tool_with()), parallel execution opt-in, stateful tools with store_as(), tool-choice control, and the default vs. explicit error-handling contract. | |
| ukgovernmentbeis-inspect-ai-agents | Covers the Inspect Agent protocol and all built-in agents: ReAct loop, Deep Agent (subagent delegation, memory, planning), multi-agent composition via handoffs and as_tool, the Agent Bridge for external frameworks, the Human Agent for baselining, and Agent Intervention (ACP) for real-time bidirectional human-agent control including operator interrupt/redirect and agent-initiated questions and notifications. Includes implementation details, configuration options, limits, and gotchas drawn directly from source and docs. | |
| ukgovernmentbeis-inspect-ai-sandboxing | Covers sandbox runtimes (local, docker, k8s, and pip-installable extensions), compose patterns for multi-service CTF-style evals, the sandbox() callable and its full method surface, file seeding, resource limits, and the tool-approval policy system (human/auto approvers, custom approvers via @approver, ToolCallViewer) that gates tool execution before it reaches any sandbox. | |
| ukgovernmentbeis-inspect-ai-logs-and-analysis | Covers the .eval log format, the EvalLog Python object, all log-reading/writing/editing APIs, the inspect view log viewer (including live view, bundling, and publishing), the inspect_ai.analysis dataframe API (evals_df, samples_df, messages_df, events_df, column groups, prepare() operations), and the Inspect Scout transcript scanner. Authoritative source for working with evaluation artifacts after a run completes. | |
| ukgovernmentbeis-inspect-ai-eval-sets | Covers eval_set / inspect eval-set for running multi-task, multi-model benchmark sweeps with automatic retry/resume mechanics, Inspect Scout scanner integration, per-sample and scoped limits (time, token, cost, message, working), early stopping, and error-handling discipline including crash recovery and cancelled-run scoring. | |
| ukgovernmentbeis-inspect-ai-cli-and-config | Complete reference for the inspect CLI: every subcommand registered in _cli/main.py, the full configuration-precedence chain (built-in default → Task arg → env var → .env file → --run-config → CLI flag → eval() kwarg), INSPECT_* environment variables, .env file loading semantics, and the -T/-M/-S task/model/solver argument shortcuts. | |
| ukgovernmentbeis-inspect-ai-extensions | The five Inspect extension points — model APIs, sandboxes, approvers, storage, and hooks — all wired through setuptools entry points so third-party packages register on equal terms with built-in providers. Covers decorator signatures, lifecycle rules, required class methods, and operational controls (INSPECT_REQUIRED_HOOKS, enabled(), API key override). | |
