terrarium-user-guide

name	terrarium-user-guide
description	Guide for running agent tasks with Terrarium. Use this skill when the user wants to configure a job, choose agents, run tasks via CLI or Python API, understand results and metrics, or troubleshoot execution issues. Activate whenever someone mentions running tasks, configuring agents, job TOML files, trial results, evaluation scores, or anything related to using Terrarium.

Terrarium User Guide

Terrarium is a Python framework for building living environments and running LLM agents through multi-stage, multi-turn tasks. Task scripts (pure Python) orchestrate capabilities like email, databases, calendars, and workspaces, while collecting structured results and metrics.

Prerequisites

uv sync                  # install dependencies

For sandbox-based agents (claude_code, openclaw, codex, hermes), build the Docker image:

docker build -t terrarium/claude-code -f docker/claude-code.Dockerfile docker/
docker build -t terrarium/openclaw -f docker/openclaw.Dockerfile docker/
docker build -t terrarium/codex -f docker/codex.Dockerfile docker/
docker build -t terrarium/hermes -f docker/hermes.Dockerfile docker/

Set environment variables in .env at the project root:

ANTHROPIC_API_KEY=sk-...
ANTHROPIC_BASE_URL=https://api.anthropic.com  # optional
OPENAI_API_KEY=sk-...
OPENAI_BASE_URL=https://api.openai.com        # optional

Two Ways to Run

Option 1: CLI

terrarium run -c job.toml

The CLI shows a live execution log, per-trial results with pass/fail checks, and aggregated metrics.

Option 2: Python API

Run a batch job:

import asyncio
from terrarium import Job, JobConfig, AgentConfig

config = JobConfig(
    agents=[AgentConfig(name="claude_code")],
    datasets=["demo/demo_dataset"],
    n_attempts=1,
)
result = asyncio.run(Job(config).run())

for tr in result.trial_results:
    print(f"{tr.trial_name}: {tr.checker_result.score}")

Run a single trial:

from terrarium import Trial, TrialConfig, TaskConfig, AgentConfig

config = TrialConfig(
    task=TaskConfig(path="demo/demo_dataset/branch_and_loop"),
    agent=AgentConfig(name="claude_code"),
)
result = await Trial(config).run()
print(result.checker_result.score)

Job Configuration (TOML)

A complete job config:

job_name = "my_eval"
datasets = ["demo/demo_dataset", "benchmarks/tau2"]
tasks = ["path/to/standalone_task"]     # optional, for tasks not in a dataset
n_attempts = 3                          # repeat each agent x task combo
n_concurrent_trials = 4                 # parallel execution slots

[[agents]]
name = "claude_code"
model_name = "claude-sonnet-4-6"        # optional, overrides default model

[[agents]]
name = "openclaw"
model_name = "terrarium/anthropic/claude-sonnet-4.6"

[retry]
max_retries = 1
min_wait_sec = 1.0
max_wait_sec = 60.0
wait_multiplier = 2.0

Equivalent Python:

JobConfig(
    job_name="my_eval",
    agents=[
        AgentConfig(name="claude_code", model_name="claude-sonnet-4-6"),
        AgentConfig(name="openclaw", model_name="terrarium/anthropic/claude-sonnet-4.6"),
    ],
    datasets=["demo/demo_dataset", "benchmarks/tau2"],
    tasks=["path/to/standalone_task"],
    n_attempts=3,
    n_concurrent_trials=4,
    retry=RetryConfig(max_retries=1),
)

Key fields

Field	Default	Description
agents	(required)	List of agent configs, each with name and optional model_name, kwargs
job_name	timestamp	Name for the output directory
datasets	[]	Paths to dataset directories (each containing dataset.toml)
tasks	[]	Paths to individual task directories (ad-hoc, no dataset)
n_attempts	1	How many times to repeat each agent x task combination
n_concurrent_trials	4	Max parallel trial executions
agent_setup_timeout_sec	None	Timeout for agent setup phase
agent_exec_timeout_sec	None	Timeout for task execution phase
retry	no retries	Retry config: max_retries, min_wait_sec, max_wait_sec, wait_multiplier

Execution expansion

The job engine expands agents x (datasets + tasks) x n_attempts into individual trials. For example, 2 agents, 1 dataset with 3 tasks, 2 attempts = 12 trials.

Agents

Built-in agents

Name	Runs in	Description
claude_code	Docker	Claude Code CLI with stream-json output
openclaw	Docker	OpenClaw CLI with session JSONL output
codex	Docker	OpenAI Codex CLI with session JSONL output
hermes	Docker	NousResearch hermes-agent CLI with session JSONL export
mini	In-process	Lightweight agent using litellm, for testing

Agent configuration

claude_code — runs Claude Code CLI in Docker (terrarium/claude-code). Requires ANTHROPIC_API_KEY in .env.

[[agents]]
name = "claude_code"
model_name = "claude-sonnet-4-6"
kwargs = { max_turns = 50 }          # optional

kwarg	type	default	description
api_key	str	from ANTHROPIC_API_KEY env	API key override
max_turns	int	None (unlimited)	Max tool-call turns per act()

openclaw — runs OpenClaw CLI in Docker (terrarium/openclaw). Requires a models config JSON file.

[[agents]]
name = "openclaw"
model_name = "terrarium/anthropic/claude-sonnet-4.6"
kwargs = { models_config_path = "./models.json" }

kwarg	type	default	description
models_config_path	str	"" (required)	Path to models provider config JSON

codex — runs OpenAI Codex CLI in Docker (terrarium/codex). Requires OPENAI_API_KEY in .env.

[[agents]]
name = "codex"
model_name = "gpt-5.4"                       # or "openai/gpt-5" via OpenRouter

kwarg	type	default	description
api_key	str	from OPENAI_API_KEY env	API key override

hermes — runs NousResearch hermes-agent CLI in Docker (terrarium/hermes). Requires a model config YAML file.

[[agents]]
name = "hermes"
model_name = "claude-sonnet-4-6"
kwargs = { model_config_path = "./hermes_model.yaml" }

kwarg	type	default	description
model_config_path	str	"" (required)	Path to hermes model config YAML

mini — in-process agent using litellm. No Docker needed. Supports tool registration and custom system prompts from task code.

[[agents]]
name = "mini"
model_name = "gpt-4o-mini"
kwargs = { temperature = 0.0, max_tool_calls = 100 }

kwarg	type	default	description
temperature	float	0.0	LLM sampling temperature
max_tool_calls	int	None (unlimited)	Max total tool calls per act()
system_prompt	str	None	System prompt (can also be set from task code)
tools	list	None	Tool callables (typically set from task code via register_tools())

Custom agent — import any class implementing BaseAgent:

[[agents]]
name = "my_agent"
import_path = "my_package.agent:MyAgent"

model_name is passed as the model kwarg to the agent constructor. All other kwargs are passed through directly.

Sandbox Providers

Capabilities and containerized agents run inside sandboxes provisioned by a SandboxProvider. The provider is selected per-job through sandbox_provider and propagates to every Trial the Job expands.

Built-in providers

Name	Backend	Notes
docker	Local Docker daemon (default)	One container per sandbox on a per-session Docker network.
k8s	Kubernetes cluster	One Pod per sandbox in a shared namespace.

Provider configuration

docker — default backend, talks to the local Docker daemon. No configuration needed for Docker Desktop / a local daemon.

[sandbox_provider]
name = "docker"
kwargs = { external_network = "shared-net" }  # optional

kwarg	type	default	description
external_network	str	None	Join an existing Docker network instead of creating a per-session one.

k8s — runs sandboxes as Pods in a Kubernetes cluster. Requires a kubeconfig the driver can use and an existing namespace.

[sandbox_provider]
name = "k8s"
kwargs = { namespace = "terrarium" }

kwarg	type	default	description
namespace	str	(required)	Namespace pods are created in. Must exist; provider does not auto-create.
kubeconfig	str	None	Path to a kubeconfig file. Falls back to KUBECONFIG env var or ~/.kube/config.
image_pull_secrets	list[str]	None	Kubernetes image pull secret names to attach to sandbox Pods.

Custom provider — implement any class extending SandboxProvider:

[sandbox_provider]
import_path = "my_package.providers:MyProvider"

kwargs is forwarded directly to the provider constructor.

Dataset Configuration

A dataset is a directory with dataset.toml and subdirectories each containing a task:

my_dataset/
  dataset.toml
  task_a/
    task.toml
    task.py
  task_b/
    task.toml
    task.py

The dataset.toml configures metrics:

[metadata]
name = "my_benchmark"

[metrics]
types = ["mean", "max", "pass@5"]   # default: ["mean"]

Tasks are discovered recursively — nested directories like domain/subdomain/task/ work.

Available metrics

Metric	Description
mean	Average score across trials
max	Highest score
min	Lowest score
sum	Total score
pass@k	Probability at least 1 of k random picks scores 1.0 (averaged per task)

Output Structure

Results are persisted as JSON:

outputs/
  {job_name}/
    config.json        # JobConfig
    result.json        # JobResult (all trials + aggregated stats)
    {trial_name}/
      config.json      # TrialConfig
      result.json      # TrialResult (trajectory, checks, timing)

Trial naming

Trial directories follow the pattern: {agent}__{model}__{source}__{task}[__attemptN]

For example: claude_code__claude-sonnet-4-6__demo__branch_and_loop__attempt0

Result viewer

Open demo/viewer.html in a browser and drag in any result.json (job-level or trial-level) to explore:

• Job overview with stats and score distribution
• Per-trial details with full trajectory (tool calls, thinking blocks)
• Timing analysis and checker results

Understanding Results

Scores

Each task defines checkers — boolean predicates that verify expected outcomes. The trial score is n_passed / n_total (0.0 to 1.0).

JobResult

A JobResult is the top-level output of a job run:

Field	Description
trial_results	List of TrialResult, one per trial execution
stats	JobStats — aggregated statistics (see below)
timing	Wall-clock start/end/duration for the entire job

TrialResult

Each trial produces a TrialResult:

Field	Description
trial_name	Identifier following {agent}__{model}__{source}__{task} pattern
task_info	Task name, path, and source (dataset name or "adhoc")
agent_info	Agent name, model, version
checker_result.score	0.0–1.0, fraction of checks passed
checker_result.checks	Individual check names, pass/fail, and tags
trajectory.messages	All messages exchanged (user instructions, agent responses, tool calls)
trajectory.metrics	Token usage (input, output, cache read/creation), LLM calls, tool calls
exception_info	Error details if the trial failed (treated as score 0.0 in metrics)
timing	Wall-clock duration for the entire trial
setup_timing	Duration for agent setup phase
execution_timing	Duration for task execution phase

JobStats

Trial results are grouped by {agent}__{model}__{source}. Each group gets an AgentDatasetStats:

Field	Description
n_trials	Total trials in this group
n_errors	Trials that raised exceptions
metrics	Computed metrics from dataset config, e.g. {"mean": 0.85, "max": 1.0}
score_stats	Score distribution — maps score to trial names, e.g. {1.0: ["trial_a"], 0.5: ["trial_b"]}
exception_stats	Error distribution — maps exception type to trial names

The top-level JobStats also has n_trials and n_errors summed across all groups.