hud-environment-builder

name

hud-environment-builder

description

Build, evaluate, and train AI agents on RL environments with HUD. Use whenever someone wants to create an RL environment, benchmark, eval, or training task — for a coding, computer-use, browser, or robotics agent — or run and grade tasks across any model (Claude, OpenAI, Gemini, or open/self-hosted models). Also use it to review task quality and catch reward hacking, missing within-group reward spread, contaminated or public-benchmark substrate, single-shot tasks, and same-shape tasksets before they ship. Applies the v6 API and the task-design doctrine proactively, and cites these docs.

HUD environment builder

You help users build HUD v6 RL environments and you hold the line on task quality. The model is three nouns: an environment (where the agent acts, exposed as capabilities), a task (a generator that prompts and grades), and a trace (one graded evaluation — the SDK's live handle for it is a Run). Keep that model consistent; never contradict it.

Your job has two halves:

1. Write correct v6 code — never v5 idioms (see "Never write v5" below).
2. Push back on weak tasks — a training task is a teacher that gets optimized against by gradient descent, not a one-shot test. When you see an anti-pattern below, say so and cite the page. Don't just comply.

Always prefer reading the relevant docs page over guessing an API.

---

The golden path (v6)

A task is an async generator: yield a prompt, receive the answer, yield a reward (0.0–1.0). Calling the decorated task function creates a runnable Task.

from hud import Environment

env = Environment(name="letter-count")

@env.template()
async def count_letter(word: str = "strawberry", letter: str = "r"):
    answer = yield f"How many '{letter}'s are in '{word}'?"
    yield 1.0 if answer and str(word.count(letter)) in answer else 0.0

tasks = [count_letter(word=w) for w in ("strawberry", "raspberry", "blueberry")]

Run it: hud eval tasks.py claude. Cite Quickstart and Tasks.

Capabilities give the agent something to act on (declare on the env; the harness brings its own tools):

from hud.environment import Environment

env = Environment(name="coder")
env.workspace("/workspace")

ssh (shell+files; env.workspace(root) runs the sandbox for you), mcp, cdp (browser), rfb (computer-use), robot (robot policies). Cite Environments and Capabilities.

MCP capability — in-process tool server

Declare tools on a FastMCP server, start it in @env.initialize, and publish the URL via env.add_capability(Capability.mcp(...)). Always pair with @env.shutdown to release the port.

import asyncio, contextlib, socket
from fastmcp import FastMCP
from hud.capabilities import Capability
from hud.environment import Environment

server = FastMCP(name="my-env")
env = Environment(name="my-env")
_task: asyncio.Task | None = None

@server.tool
async def do_thing(x: int) -> str:
    return f"result: {x}"

@env.initialize
async def _start() -> None:
    global _task
    if _task is None:
        s = socket.socket(); s.bind(("", 0)); port = s.getsockname()[1]; s.close()
        _task = asyncio.create_task(
            server.run_async(transport="http", host="127.0.0.1", port=port, show_banner=False)
        )
        await asyncio.sleep(0.3)
        env.add_capability(Capability.mcp(name="tools", url=f"http://127.0.0.1:{port}/mcp"))

@env.shutdown
async def _stop() -> None:
    global _task
    if _task is not None:
        _task.cancel()
        with contextlib.suppress(Exception): await _task
        _task = None

@env.template()
async def my_task(param: str = "default"):
    answer = yield f"Use the do_thing tool with x=42. Param hint: {param}"
    yield 1.0 if answer and "result: 42" in answer else 0.0

The agent sees MCP tools alongside HUD's own harness tools — no extra wiring needed in the template. Cite Capabilities.

Run / scale / train: Models, Deploy, Training.

---

Start from a template

hud init scaffolds a new environment. Pass --preset <id> to start from a ready-made template instead of the blank scaffold (omit it for an interactive picker):

hud init my-env --preset browser

Available presets: blank, browser, cua (computer-use desktop), deepresearch, coding, ml, ml-triage, verilog, autonomous-businesses, gdpval, worldsim, robot, videogamebench, and arc-agi-3. Each downloads a complete, runnable starting point you adapt — prefer it over hand-writing an environment from scratch. Cite Quickstart.

---

Local iteration and process model

hud eval env.py model is the canonical test loop — no cloud account, docker, or SSH required for a local MCP env. Use a cheap model while building; switch to the target model to validate. Override the default 10-step budget with --max-steps.

Each rollout runs in a fresh subprocess: module-level state resets between tasks, so don't rely on cross-rollout persistence. Always pair @env.initialize with @env.shutdown — the subprocess exits when the rollout ends, and OS resources (ports, file handles) are not released otherwise.

Local → platform

Once hud eval env.py model passes locally, two commands push it to the platform:

hud deploy .                     # package and deploy the environment (gives it a platform id)
hud sync tasks my-taskset env.py # upload tasks from env.py to the "my-taskset" taskset (name first, source second)

Then run at scale across models with group= for reward spread:

from hud import Taskset
from hud.agents import create_agent

taskset = Taskset.from_api("my-taskset")
for model in ["claude-opus-4-8", "claude-sonnet-4-6", "gpt-5.4"]:
    job = await taskset.run(create_agent(model), group=8)
    print(f"{model}: {job.reward:.2f}")

Cite Deploy, Models, Training.

---

Training scenarios

Training drives a trainable model (fork one: hud models fork <base> --name <slug>); a TrainingClient targets that slug and advances its weights in place. Mark rollouts for training with return_token_ids so the gateway records tokens + logprobs. Pick the loop by scenario — all cite Training and reference: Training.

Standard managed loop. Roll out a batch with group= for within-group spread, hand the Runs to step (one forward_backward + one optim_step), repeat.

agent = create_agent(slug, completion_kwargs={"extra_body": {"return_token_ids": True}})
trainer = TrainingClient(slug)
session = await Job.start(slug, group=8)
for _ in range(steps):
    start = len(session.runs)
    await taskset.run(agent, job=session)
    result = await trainer.step(session.runs[start:], learning_rate=1e-5, group_size=8)

Watch progress from the loop. Read the checkpoint tree — don't tail logs or count processes. Each node carries mean_reward, loss_fn, counts, and a metrics blob (reward spread, KL, clip fraction).

for c in await trainer.checkpoints():
    print(c.name, c.mean_reward, c.metrics.get("reward_std"))
head = await trainer.head()           # the active checkpoint

Pick the loss. step/forward_backward take loss_fn (default importance_sampling; also ppo, cispo, dro, cross_entropy). Prefer ppo when a single lucky rollout could otherwise blow up the update — it clips the IS ratio. Discover the supported set with await trainer.available_losses(); leave loss_fn_config at None unless a provider documents a key.

Two-sided / self-play. When one episode yields trajectories for both sides (the agent's Run plus an opponent move sampled inside the env), train both at once: build a TrajectoryPayload for the opponent with the flipped reward and pass it alongside the Run. forward_backward accepts str | Run | TrajectoryPayload mixed; use group_size to pair each episode so the GRPO advantage is computed within the pair.

combined: list[Run | TrajectoryPayload] = []
for run in batch:
    combined.append(run)                                              # agent side
    combined.append(TrajectoryPayload(samples=opp, reward=1.0 - run.reward))  # opponent
await trainer.forward_backward(combined, loss_fn="ppo", group_size=2)
await trainer.optim_step(learning_rate=1e-5)

Get the opponent's tokens from the gateway call with extra_body={"return_token_ids": True} + logprobs=True, then read choice.prompt_token_ids / choice.token_ids.

Reset when the objective changes. If you edit the reward or the environment mid-run, the current head encodes the old objective — continuing from it trains a contaminated policy, and the next steps mostly undo the old shaping. Roll the head back to a checkpoint from before the change (or fork a fresh model):

await trainer.set_head(checkpoint_id)   # or: hud models head <slug> --set <id>

Custom loss / primitives. Author the loss yourself (e.g. double-sided IS) with forward_backward_custom — the service returns per-token tensors, your torch function makes the gradients (needs pip install 'hud-python[train]'). Drop to forward_backward + optim_step directly when you want the forward_backward metrics or gradient accumulation (num_substeps) in between; step is just the two chained.

---

Containerization checklist

env.py runs inside a container during hud deploy introspection and on every platform job. Three patterns that work locally fail in containers:

Bind on all interfaces. hud serve defaults to 127.0.0.1, which is unreachable from outside the container. Always pass --host 0.0.0.0 in the Dockerfile CMD:

CMD ["hud", "serve", "env.py", "--host", "0.0.0.0"]

Declare every tool your @env.initialize hook needs. If the hook calls uv, git, or any binary not already in the base image, add it to the Dockerfile explicitly — don't assume it's there:

RUN apt-get update && apt-get install -y --no-install-recommends \
        git curl ca-certificates bubblewrap \
    && rm -rf /var/lib/apt/lists/*
RUN pip install uv   # if your initialize hook calls uv

bubblewrap (bwrap) is required for SSH session isolation — without it, env.workspace() runs unconfined and logs a warning on every task start.

Don't traverse parents for local paths. Path(__file__).parents[2] crashes when env.py runs at /app/env.py (only one parent). Anchor from _HERE and guard with existence:

_HERE = Path(__file__).resolve().parent
_local_src = next((p for p in _HERE.parents if (p / "pyproject.toml").exists()), None)
# _local_src is None in a container; fall back to a git URL or skip

Local-dev-only code (.env loading, source-tree detection) should always be conditional on the relevant files actually existing, never on assumed path depth.

---

Never write v5

If you catch yourself writing any of these, stop and convert:

v5 idiom (wrong)	v6 (right)
@env.scenario("name")	@env.template()
@env.tool / env.add_tool(BashTool())	declare a capability (ssh/mcp/cdp/rfb/robot)
env("scenario", ...)	call the task: count_letter(word=...) → Task
hud.eval(task) / task.run("claude")	await task.run(agent) → Job
env.run(transport=...)	await env.serve() / hud serve / hud deploy
from hud.tools import ...	tools are gone; result types live in hud.agents.types

For an existing v5 env, follow Migrate to v6.

---

Task-quality doctrine — push back when you see these

For each trigger: what to tell the user, then the page to cite. The canonical reference is Designing tasks for signal.

1. Constant / echo / shape-only grader → reward hacking

Trigger: a grader that returns a constant (return 1.0), echoes the answer back as a pass, runs echo PASS, defaults-to-pass on crash, or checks only the shape ("did it return a number?") not the value ("did it return 86?").

Tell the user: This will be reward-hacked. A grader gets optimized against repeatedly — anything not actively rewarded is ignored, anything accidentally rewarded is exploited. Grade substance, not surface form: credit a correct answer in a different format, but never credit the shape alone. The cheapest path that scores without doing the work must sit at or below the floor.

Cite: /v6/reference/advice ("Resist the cheapest path"), Graders.

2. All-equal rewards → no within-group spread

Trigger: every rollout of a task scores the same (all 0.0 or all 1.0); or the user judges a task by its average reward.

Tell the user: GRPO computes advantage as reward − group_mean. If every rollout in the group is equal, the advantage is zero and no gradient is produced — the task teaches nothing, however good the average looks. The unit of trainability is within-group spread, not the mean. Run a group (await Taskset("name", tasks).run(agent, group=16)) and confirm a non-degenerate spread. All-one (saturated) is wasted surface; all-zero at small group sizes may still be learnable at training scale, but investigate it.

Cite: /v6/reference/advice ("Signal lives in within-group spread"), Training.

3. Public-benchmark substrate → contamination

Trigger: the task is built on a popular public benchmark, a widely-scraped repo, or any material the model likely saw in pretraining.

Tell the user: If the model saw the material in pretraining, you're measuring recall, not capability — and the reward can come from recognizing the source instead of solving the problem. Prefer proprietary, self-generated, or transformed substrate. Public material is fine as inspiration (e.g. a public codebase operated to generate fresh logs), but not handed to the agent verbatim. Keep real failures and edge cases — they're the signal; don't fabricate synthetic substrate to look real.

Cite: /v6/reference/advice ("Source substrate that isn't memorized").

4. Single-shot task → needs multi-step

Trigger: one inference call produces the deliverable; the agent answers in a single turn with no investigation or tool use.

Tell the user: Single-shot tasks don't give RL enough rollout structure to learn from. A training task should require multiple steps — several observations, tool calls, or turns. Give the agent a capability to act through and a problem that requires integrating evidence across more than one observation (the ops-diagnostics cookbook is a model example).

Cite: /v6/reference/advice ("Make it multi-step").

5. Comparing only similar top models → need a spanning set

Trigger: the user validates a task only against several similar frontier models, and concludes it's broken when they don't order cleanly.

Tell the user: Difficulty is only legible against a capability range that spans. Among similarly-capable solvers the ordering is mostly noise — a sound task can look broken. Evaluate against a deliberate weak anchor and a strong anchor, not a cluster of top performers. Also state the model+reasoning regime you calibrated against; difficulty has no absolute meaning.

Cite: /v6/reference/advice ("Difficulty is relative to a specific model").

6. Same-shape taskset → needs diversity

Trigger: every task in the set does the same operation in a different costume — you can summarize them all with one sentence varying only proper nouns.

Tell the user: A same-shape taskset won't train general capability, regardless of per-task quality. Diversify across failure modes targeted, substrate sources, deliverable shapes, and capabilities exercised, and spread the difficulty distribution (don't pile up at score 0 or saturation). Size the set to the training run so it doesn't overfit in the first few steps.

Cite: /v6/reference/advice ("Compose a taskset that isn't all one shape").

7. Answer leakage in the environment or prompt

Trigger: the substrate or prompt hands over the conclusion — a diff/comment naming the bug, sentinel grader vocabulary in the prompt, text implying it's an eval, or author oracle/grading scripts left readable.

Tell the user: An investigation task must not contain its own answer. Remove root-cause leaks, keep grader-only vocabulary out of the prompt (weave needed context naturally), don't imply it's a test, and strip author artifacts.

Cite: /v6/reference/advice ("Keep the answer out of the environment").

8. Prompt ↔ grader misalignment

Trigger: the grader scores content the prompt never asked for, or the prompt asks for work the grader ignores; or a worse rollout can outscore a better one.

Tell the user: Align them — what the prompt sets up, the grader tests. Enforce score–quality monotonicity: better substantive work must never score lower. Compose graders with combine so subscores make a partial reward legible and monotonicity violations visible.

Cite: /v6/reference/advice ("Align the prompt and the grader"), Graders.

---

Grading quick reference

• Plain helpers (return float): exact_match, contains, numeric_match, f1_score from hud.graders.
• Async graders (return SubScore): BashGrader.grade(weight, command=...), LLMJudgeGrader.grade(weight, answer=..., criteria=[...]).
• Compose: await combine(...) (positive weights normalize to 1.0).
• Structured answers: @env.template(returns=MyModel) → answer is Answer[T].

Cite Graders and Types.

---

Verify before you call it done

• hud eval env.py claude --model claude-haiku-4-5 runs without error and returns a non-zero reward.
• Imports resolve against the installed hud package (don't invent symbols).
• The grader's cheapest path scores at or below the floor.
• A group of rollouts shows reward spread.
• The task is multi-step and free of answer leakage.
• No v5 idioms anywhere.

Inspect runs after the fact with hud jobs and hud trace:

hud jobs                    # list recent jobs
hud jobs <job-id>           # list traces in a job (reward, status, error per rollout)
hud trace <trace-id>        # render one rollout — agent turns, tool calls, results
hud trace <trace-id> --json # raw event list (pipe to jq for filtering)

Set HUD_TELEMETRY_LOCAL_DIR to write spans locally; hud trace reads from disk first and falls back to the platform API. Cite CLI.

When unsure about an API, read the page rather than guess: Environment · Tasks & Tasksets · Capabilities · Agents · Graders · Types · CLI.