i4h-workflow-validate

name	i4h-workflow-validate
version	0.6.1
description	Validate, evaluate, or run i4h envs. Use for policy/checkpoint rollouts and scripted state-machine smoke runs.
license	Apache-2.0
metadata	{"author":"Isaac for Healthcare Team <[e-mail]>","tags":["isaac-for-healthcare","i4h","agentic-workflow","validation","policy-rollout"]}

i4h Workflow — Validate

Purpose

Roll out a policy or scripted state-machine controller against an env and record verification episodes to an HDF5. Use when the user asks to validate, evaluate, run, or rollout a policy/checkpoint, or asks for surgical state-machine smoke runs.

Base Code

These steps drive the i4h-workflows base code (the workflows/agentic/ tree). To reuse an existing checkout, set I4H_WORKFLOWS to its path (no clone happens). Otherwise this resolves the current repo, or clones to ~/i4h-workflows — pick that default without prompting. Run every command below from the resolved root:

# Resolve the i4h-workflows base code (provides workflows/agentic/).
ROOT="${I4H_WORKFLOWS:-$(git rev-parse --show-toplevel 2>/dev/null)}"
if [ ! -d "$ROOT/workflows/agentic" ]; then
  ROOT="${I4H_WORKFLOWS:-$HOME/i4h-workflows}"
  [ -d "$ROOT/workflows/agentic" ] || git clone https://github.com/isaac-for-healthcare/i4h-workflows "$ROOT"
fi
export I4H_WORKFLOWS="$ROOT"; cd "$ROOT"

Basics

• Env config (source of truth): workflows/agentic/config/environments/<env>.yaml — read it for the <env> defaults: policy.model_repo/model_revision, policy.task_description, policy.health_port, and arena.max_timesteps.
• Validation runs the policy daemon and Arena together; both processes are required.
• The policy daemon is headless. Arena opens the sim window by default; add --headless --enable_cameras --rendering_mode performance only when the user explicitly asks for headless/no-window execution.
• In Claude Code --print, Codex exec, or any other non-interactive/fresh session, policy evaluation must use Step 2A as one foreground bash command. Do not start the policy and Arena in separate tool calls, do not use Claude background tasks for eval, and do not return to the user until Arena exits and the policy cleanup has run.
• In Claude Code specifically, do not use the Bash tool's background mode for Evaluate ... prompts, do not launch a command ending in &, and do not say "the eval is running in the background." The answer is not complete until the HDF5/log summary has been inspected.
• README quick-run prompts that say "with the state machine" use Arena --state-machine and do not start a policy daemon.
• Do not run the VLM annotator unless the user asks for success labels.
• assemble_trocar is inference-only — validate its YAML default model or a compatible N1.5 checkpoint.

Inputs

• ENV_ID: env YAML id.
• EPISODES: 1 for sanity, more for real eval.
• MAX_TIMESTEPS: use the user-requested cap when the prompt gives one (for example, 300 timesteps -> MAX_TIMESTEPS=300); otherwise read arena.max_timesteps from the env YAML for normal evaluation. Use 200 only when the user explicitly asks for a smoke, sanity, or quick check.
• MODEL_PATH (optional): path to a checkpoint-NNNN/ directory containing model-0000{N}-of-*.safetensors, experiment_cfg/, and processor/. Omit to use YAML policy.model_repo.
• USE_LATEST_CHECKPOINT=1: set this when the prompt says "new checkpoint" or "latest checkpoint" and MODEL_PATH is not already known.
• STATE_MACHINE: true only when the prompt explicitly says state machine.

Run

Run the steps below in order with the bash tool. Script paths like policy/run.sh, arena/run.sh, and stop.sh are commands inside bash, not tool names.

For policy/checkpoint evaluation in Claude Code --print, Codex, codex exec --ephemeral, or any other non-interactive fresh session, use Step 2A after setup. Background policy daemons launched by a finished shell can be cleaned up before Arena connects; the controlled shell keeps policy and Arena in one process lifetime and always stops the daemon afterward. In an interactive local-agent tmux session, the separate Step 2 / Step 3 / Step 4 flow is also acceptable.

Step 1 — setup

REPO_ROOT="${I4H_WORKFLOWS:-$(git rev-parse --show-toplevel 2>/dev/null)}"; [ -d "$REPO_ROOT/workflows/agentic" ] || REPO_ROOT="$HOME/i4h-workflows"
ENV_ID=scissor_pick_and_place
EPISODES=1
ENV_CONFIG="${REPO_ROOT}/workflows/agentic/config/environments/${ENV_ID}.yaml"
[ -f "${ENV_CONFIG}" ] || { echo "missing env config: ${ENV_CONFIG}" >&2; exit 1; }
PYTHON="${REPO_ROOT}/workflows/agentic/arena/.venv/bin/python"
[ -x "${PYTHON}" ] || PYTHON="${REPO_ROOT}/workflows/agentic/.venv/bin/python"
[ -x "${PYTHON}" ] || { echo "missing workflow python env; run i4h-workflow-setup first" >&2; exit 1; }
MAX_TIMESTEPS="${MAX_TIMESTEPS:-$("${PYTHON}" -c 'import sys, yaml; print(yaml.safe_load(open(sys.argv[1], encoding="utf-8"))["arena"]["max_timesteps"])' "${ENV_CONFIG}")}"
RUNS_ROOT="${REPO_ROOT}/workflows/agentic/runs"

# For prompts such as "Run eval using new checkpoint for 300 timesteps":
#   set MAX_TIMESTEPS=300 and USE_LATEST_CHECKPOINT=1 before this block.
if [ "${USE_LATEST_CHECKPOINT:-0}" = "1" ] && [ -z "${MODEL_PATH:-}" ]; then
  MODEL_PATH="$(find "${RUNS_ROOT}" -path '*/checkpoint/checkpoint-*' -type d -printf '%T@ %p\n' 2>/dev/null | sort -nr | head -1 | cut -d' ' -f2-)"
  [ -n  ] || {   >&2;  1; }


RUN_DIR=
 -p  
 -sfn  

Step 2 — policy daemon

Skip this step when STATE_MACHINE=true. For Codex/non-interactive sessions, prefer Step 2A instead of this separate policy-daemon step.

POLICY_ARGS=(--env "${ENV_ID}" --ensure --log "${RUN_DIR}/logs/policy.log")
[ -n "${MODEL_PATH:-}" ] && POLICY_ARGS+=(--model-path "${MODEL_PATH}")
"${REPO_ROOT}/workflows/agentic/policy/run.sh" "${POLICY_ARGS[@]}"

Run this command exactly as a normal foreground bash command. Do not pipe it to head, cat, tee, or tail; do not add a separate stop, background launch, sleep, grep loop, curl check, or docker ps. policy/run.sh --ensure owns reuse, stop/restart, and start-and-ready behavior.

Step 2A — controlled policy rollout

Use this instead of separate Step 2 / Step 3 / Step 4 when running in Claude Code --print, Codex, codex exec --ephemeral, or another fresh non-interactive session. Run it as a normal foreground bash command; do not put it in the background and do not answer until it prints ARENA_STATUS.

POLICY_ARGS=(--env "${ENV_ID}" --ensure --log "${RUN_DIR}/logs/policy.log")
[ -n "${MODEL_PATH:-}" ] && POLICY_ARGS+=(--model-path "${MODEL_PATH}")

cleanup_policy() {
  "${REPO_ROOT}/workflows/agentic/stop.sh" policy --env "${ENV_ID}" >/dev/null 2>&1 || true
}
trap cleanup_policy EXIT

"${REPO_ROOT}/workflows/agentic/policy/run.sh" "${POLICY_ARGS[@]}"
ARENA_STATUS=0
"${REPO_ROOT}/workflows/agentic/arena/run.sh" --env "${ENV_ID}" \
  --episodes "${EPISODES}" \
  --max-timesteps "${MAX_TIMESTEPS}" \
  --max-attempts 1 \
  --record-to "${RUN_DIR}/data/verify.hdf5" \
  > "${RUN_DIR}/logs/arena.log" 2>&1 || ARENA_STATUS=$?
"${REPO_ROOT}/workflows/agentic/stop.sh" policy --env "${ENV_ID}"
trap - EXIT
echo "ARENA_STATUS=${ARENA_STATUS}"

After this block, skip directly to Step 5.

Step 3 — arena rollout

Skip this step if Step 2A was used.

For state-machine smoke runs, use this Arena command instead of the policy rollout command:

"${REPO_ROOT}/workflows/agentic/arena/run.sh" --env "${ENV_ID}" \
  --state-machine \
  --episodes "${EPISODES}" \
  --max-timesteps "${MAX_TIMESTEPS}" \
  --record-to "${RUN_DIR}/data/verify.hdf5" \
  > "${RUN_DIR}/logs/arena.log" 2>&1

For policy or checkpoint evaluation:

"${REPO_ROOT}/workflows/agentic/arena/run.sh" --env "${ENV_ID}" \
  --episodes "${EPISODES}" \
  --max-timesteps "${MAX_TIMESTEPS}" \
  --max-attempts 1 \
  --record-to "${RUN_DIR}/data/verify.hdf5" \
  > "${RUN_DIR}/logs/arena.log" 2>&1

Step 4 — stop policy

Skip this step when STATE_MACHINE=true or when Step 2A was used.

"${REPO_ROOT}/workflows/agentic/stop.sh" policy --env "${ENV_ID}"

Step 5 — summarize logs

grep -E "policy job complete|run complete|Traceback|Error|FAILED" "${RUN_DIR}/logs/arena.log" || tail -80 "${RUN_DIR}/logs/arena.log"
grep -E "policy ready|Traceback|Error|FAILED" "${RUN_DIR}/logs/policy.log" || tail -30 "${RUN_DIR}/logs/policy.log"

Notes

• Launch the policy daemon with policy/run.sh --ensure, then launch Arena.
• In non-interactive Codex runs, keep the policy daemon and Arena in one controlled shell with Step 2A so the daemon is not cleaned up between tool calls.
• Once Arena exits — whether episodes succeeded or failed — shut down the policy daemon. It does not self-terminate, so leaving it running leaks GPU memory and holds its health port. Stop it with "${REPO_ROOT}/workflows/agentic/stop.sh" policy --env "${ENV_ID}".
• --record-to must be absolute. The recorder resolves relative paths against workflows/agentic/arena (its CWD) and produces a nested orphan dir.
• --max-attempts defaults to 1 for locomanip-family envs.

Optional Annotation

Run only on request:

"${REPO_ROOT}/workflows/agentic/annotator/run.sh" \
  --env "${ENV_ID}" \
  --output "${RUN_DIR}/annotations.jsonl" \
  offline \
  --hdf5-path "${RUN_DIR}/data/verify.hdf5"

Verify

• verify.hdf5 exists under ${RUN_DIR}/data/.
• Arena log shows run complete: N/M episodes succeeded.
• Policy log contains no Traceback.

Prerequisites

• Workflow set up via [[i4h-workflow-setup]] (.venv present); the policy/run.sh and arena/run.sh launches depend on it.
• An ENV_ID matching an env YAML id.
• A model source: either the env YAML policy.model_repo default, or a MODEL_PATH pointing at a checkpoint-NNNN/ dir (model-0000{N}-of-*.safetensors, experiment_cfg/, processor/).

Limitations

• Both the policy daemon and Arena are required; the daemon is headless and Arena opens the sim window unless the user explicitly asks for headless/no-window execution.
• assemble_trocar is inference-only — validate its YAML default model or a compatible N1.5 checkpoint.
• --record-to must be absolute; relative paths resolve against workflows/agentic/arena and produce a nested orphan dir.
• The VLM annotator is optional and run only on request; it is not part of the default rollout.

Troubleshooting

• Error: .venv / import fails or run.sh missing - Cause: workflow not set up. Fix: run [[i4h-workflow-setup]] first.
• Error: policy log shows Traceback / Error / FAILED before policy ready - Cause: the policy daemon failed to start (e.g. bad model source). Fix: inspect ${RUN_DIR}/logs/policy.log; verify ENV_ID / MODEL_PATH.
• Error: Arena starts before the daemon is ready - Cause: launch order. Fix: launch the policy daemon first and wait for policy ready, then launch Arena.
• Error: verify.hdf5 lands in a nested orphan dir - Cause: relative --record-to. Fix: pass an absolute path under ${RUN_DIR}/data/.
• Error: PermissionError on /data/verify.hdf5 - Cause: RUN_DIR was unset when Arena ran (setup was skipped or run out of order). Fix: run the setup lines first so RUN_DIR exists before --record-to.

Final Response

Report env, model source, episodes saved vs requested, HDF5 path, log paths.