Operator/host side of running the NVIDIA "Build an Agent" DevX workshop inside an OpenShell/NemoClaw sandbox. In the NemoClaw community repo this is the PRIMARY workshop-setup entry point: use it for "set up / run the Build-an-Agent workshop" requests when YOU are on the sandbox HOST (outside the sandbox — the machine running the OpenShell gateway + docker, e.g. via Claude Code) against this example's own vendored NemoClaw deployment (brought up by its scripts/bring-up.sh). It stages/applies the egress policy (PyPI, NIM /v1/ranking, scoped GitHub clone), stages this example's skills into the agent's skill library, optionally stages the NVIDIA API key, kicks the in-sandbox agent (which runs the `setup-workshop-nemoclaw` skill: clone + venv + JupyterLab), then opens the inbound path (openshell forward service + SSH/Teleport port-forward) so the user can open the JupyterLab token URL. Also covers sandbox lifecycle pitfalls (container restarts are only token-window-safe, recreate wipes state) and egress-denial de
Operator/host side of running the NVIDIA "Build an Agent" DevX workshop inside an OpenShell/NemoClaw sandbox. In the NemoClaw community repo this is the PRIMARY workshop-setup entry point: use it for "set up / run the Build-an-Agent workshop" requests when YOU are on the sandbox HOST (outside the sandbox — the machine running the OpenShell gateway + docker, e.g. via Claude Code) against this example's own vendored NemoClaw deployment (brought up by its scripts/bring-up.sh). It stages/applies the egress policy (PyPI, NIM /v1/ranking, scoped GitHub clone), stages this example's skills into the agent's skill library, optionally stages the NVIDIA API key, kicks the in-sandbox agent (which runs the `setup-workshop-nemoclaw` skill: clone + venv + JupyterLab), then opens the inbound path (openshell forward service + SSH/Teleport port-forward) so the user can open the JupyterLab token URL. Also covers sandbox lifecycle pitfalls (container restarts are only token-window-safe, recreate wipes state) and egress-denial debugging via the OCSF audit log. NOT for working inside the sandbox, and NOT for bare-metal/Brev/AI-Workbench installs (that installer — `setup-workshop` — ships with the workshop repo itself, not with this example).
disable-model-invocation
false
user-invocable
true
setup-workshop-nemoclaw-operator (host side)
Prepares an OpenShell/NemoClaw sandbox so the in-sandbox agent can stand up
the Build-an-Agent workshop, then opens the access path from the user's laptop
to the resulting JupyterLab. This is one half of a two-skill pair:
In the NemoClaw community repo, yes — this is the default setup path.
This example exists to run the workshop on its own NemoClaw deployment
(vendored from the developer-community-chief-of-staff recipe), so a
generic "set up the build-an-agent workshop" request lands here. The
prerequisite is a deployed sandbox: if docker ps shows no
openshell-<sandbox>-… container, deploy first (this example's
scripts/bring-up.sh, after filling .env per the README) — this skill
configures an existing deployment, it does not create one.
Not this skill's territory: installing the workshop OUTSIDE a sandbox (bare
metal on Brev, local install via AI Workbench, GPU hosts). That installer —
the setup-workshop skill — ships with the upstream workshop repo, not with
this example; point the user there.
Which side am I on?
On the host (use this skill): docker ps shows an
openshell-<sandbox>-… container, the openshell CLI is on PATH, and you are
NOT user sandbox. Inside the sandbox (use the other skill): whoami →
sandbox, /sandbox/ exists, no docker/sudo.
Conventions
SANDBOX=hermes-direct # the sandbox name (adjust)# Its container — exact, fail-closed selection by OpenShell runtime labels# (substring grep can match a similarly named sandbox, e.g. hermes-direct-2):
C=$(docker ps --filter 'label=openshell.ai/managed-by=openshell' \
--filter "label=openshell.ai/sandbox-name=$SANDBOX" --format '{{.Names}}')
[ "$(printf '%s\n' "$C" | grep -c .)" -eq 1 ] || echo"FATAL: not exactly one container for '$SANDBOX': ${C:-none}"# The deployment (scripts + policy.yaml + .env) and the workshop skills both# live in THIS example:
EXAMPLE=<nemoclaw-community>/examples/recipes/nvidia/agentic-ai-learning-path
cd"$EXAMPLE"
Policy ownership: the deployment's policy.yaml template (this example's
root) is the sandbox-create input — this flow does NOT edit it. The workshop
policy lives in the sandbox's live policy: Phase 1 applies live + the
additions from references/policy-blocks.md, and the live policy is
thereafter the source of truth. Captures
(openshell policy get "$SANDBOX" --full | sed '1,/^---$/d') are scratch
artifacts — regenerate on demand, do not track them. Consequence: a recreate
through the deployment's own machinery (scripts/bring-up.sh /
scripts/03-sandbox.sh) re-renders the STOCK template, silently reverting
every workshop grant (network AND filesystem) and wiping the workshop
filesystem — after such a recreate, re-run Phase 1, then 1b, then 2b/3 (all
idempotent).
If you are an agent (e.g. Claude Code) driving this: egress-widening
openshell policy set runs are typically denied by the permission layer
unless the human runs them or explicitly names the exact hosts being opened.
That denial is correct. Your job is to stage and verify the policy file,
then hand the human one copy-paste command with a precise "this opens
exactly: …" description. Same for the secrets write (it moves a credential).
Phase 0 — Discover state (all read-only)
docker ps --format '{{.Names}}\t{{.Status}}' | grep "$SANDBOX"# container up?
openshell policy get "$SANDBOX" | head -5 # live revision + hash
docker exec"$C"ls -la /sandbox/workshop-build-an-agent 2>&1 | head -3 # repo cloned?
docker exec"$C"ls -l /sandbox/workshop-build-an-agent/secrets.env 2>&1 # key staged?
(openshell policy get prints Active: 0 next to Status: Loaded for a
policy that has never been superseded — normal for a fresh sandbox, not "no
policy active".)
Then extract the live policy and establish which way drift runs — do NOT
assume the sandbox is running the recipe's stock policy. --full prepends a
metadata header (Version:/Hash:/… plus a --- line) that must be
stripped before the YAML is usable:
openshell policy get "$SANDBOX" --full | sed '1,/^---$/d' > /tmp/live.yaml
python3 -c 'import yaml,sys
a,b=[sorted(yaml.safe_load(open(f))["network_policies"]) for f in sys.argv[1:]]
print("template-only:", [k for k in a if k not in b])
print("live-only: ", [k for k in b if k not in a])' policy.yaml /tmp/live.yaml
Read the result in BOTH directions. template-only blocks = the live policy
is missing stock grants (unusual — investigate before overwriting them).
live-only blocks = the workshop additions are already applied (the expected
state after Phase 1, or after a prior run) → Phase 1's apply is a no-op.
Either way the template stays untouched — it is the recipe's file, and a
recreate from it always reverts to stock; Phase 1b and the lifecycle notes
below deal with that.
docker exec is fine for these filesystem peeks. It is NOT a valid
probe for egress/syscall behavior — exec'd processes bypass the per-process
Landlock/seccomp layers and yield false "allowed" results. Egress tests go
through openshell sandbox exec only (single-line commands; it rejects
multi-line args).
Phase 1 — Egress policy (one-time)
The sandbox denies all egress by default, and the deployment's stock policy
(this example's policy.yaml) does not include any workshop route — but do not assume stock is
what is live: go by the Phase 0 drift check. Blocks missing from live → apply
the additions below (live capture + additions). Blocks already live → skip
the apply; the live policy is the source of truth and no repo file needs
syncing. Exact YAML in references/policy-blocks.md:
Block
Opens
Needed for
github_git_clone
git smart-HTTP on github.com, scoped to the one workshop repo, for the git binaries
cloning the repo (skip if already cloned)
pypi_install
read-only GET to pypi.org + files.pythonhosted.org
uv pip install of the workshop deps
nvidia_retrieval
POST /v1/retrieval/** on ai.api.nvidia.com
modules 2/3 NVIDIARerank — ⚠️ the legacy /v1/ranking rule on integrate.api.nvidia.com does NOT cover llama-nemotron-rerank-1b-v2
tavily_search
POST /search+/extract on api.tavily.com
module-1 docgen, module-2 local-MCP web search, module-5 search (key: TAVILY_API_KEY)
langsmith_api
all methods on api.smith.langchain.com
module-3 eval/tracing AND silencing tracing-retry spam in every notebook (variables.env turns tracing on globally; key: LANGSMITH_API_KEY)
tiktoken_encodings
GET /encodings/** on openaipublic.blob.core.windows.net
module-7 harness_lab (tiktoken BPE download at first use)
npm_install
read-only GET to registry.npmjs.org (binary: /usr/local/bin/node)
module-5 "Deep Agents Client" tile — demo/ needs npm install or the tile only serves its "setup required" page; also fetches the mcp-remote transport for the block below
mcp_tavily
GET/POST/DELETE on mcp.tavily.com (binary: /usr/local/bin/node)
module-2 PART 2A remote MCP — the shipped default. ⚠️ Policy alone is not enough: the MCP stdio transport drops the proxy env, so the in-sandbox tune_remote_mcp_env.py is also required
/dev/pts fs grant
rw on the devpts filesystem (PTY allocation) — under filesystem_policy, not network_policies
JupyterLab's Terminal tile (terminado → pty.fork); without it the tile pops "Launcher Error: Unhandled error"
Not needed: build.nvidia.com (notebook prose only — every model call goes to
integrate.api.nvidia.com), torch/conda mirrors, npm.
⚠️ The supervisor parses filesystem_policy ONCE at container boot —
openshell policy set hot-reloads network rules but NOT filesystem grants
(verified live on OpenShell v0.0.53 AND v0.0.96: after applying a /dev/pts
grant, new spawns still built the old ruleset). Include the fs grants in the
Phase 1 apply anyway (they ride along dormant), then boot them with the
Phase 1b token-TTL-guarded restart below.
Workflow (details + YAML in the reference):
Capture the live policy WITH the metadata header stripped (raw --full
output is not valid apply input; already done if you ran the Phase 0
drift check):
openshell policy get "$SANDBOX" --full | sed '1,/^---$/d' > /tmp/live.yaml
Build the apply file = live policy + the new blocks and nothing else:
python3 scripts/build-workshop-policy.py /tmp/live.yaml /tmp/apply.yaml
(composes the additions, structurally self-verifies, idempotent — exits
non-zero on any surprise). The filesystem_policy additions (/dev/pts
rw, /sys/fs/cgroup ro) ride along dormant until Phase 1b boots them.
Do NOT edit the deployment recipe's files: the applied live policy is the
only carrier of workshop state.
Apply (human runs it if the permission layer blocks you):
openshell policy set"$SANDBOX" --policy <file> --wait# success: "✓ Policy version N submitted … loaded (active version: N)"
⚠️ policy set REPLACES the entire policy document — it does not
merge. Applying a partial/stale file silently revokes whatever it omits
(this exact mistake once reverted a GitHub grant here). OpenShell ≥ 0.0.53
also has openshell policy update for incremental adds — prefer it for
one-block changes if available.
PTY-OK → skip this phase. Denied → boot the grants with a container
restart. A restart is safe ONLY while the sandbox's bootstrap token is
valid: the token is written once at create
(/etc/openshell/auth/sandbox.jwt) and never refreshed on disk, so the
supervisor re-reads it on boot — valid token → clean recovery (verified on
OpenShell v0.0.96: Ready again in ~10 s, fs grants live); expired token →
Unauthenticated/ExpiredSignature crash loop, sandbox stuck in
Provisioning (verified live on v0.0.53-created and v0.0.96-resumed
sandboxes). Guard on container age and fail closed:
CREATED=$(docker inspect "$C" --format '{{.Created}}')
AGE=$(( $(date +%s) - $(date -d "$CREATED" +%s) ))
if [ "$AGE" -lt 2700 ]; then# 45-min guard against the ~1 h token TTL
docker restart "$C"elseecho"sandbox older than the token-safe window — use the recipe-recreate fallback below"fi
In the normal quickstart flow the sandbox is minutes old, so the guard
always passes. After the restart: wait for openshell sandbox list to show
Ready (seconds), re-resolve $C (label selection, as in Conventions),
re-probe PTY (expect PTY-OK).
A container restart does NOT relaunch the agent stack (nemoclaw-start).
Relaunch it with the create-time authorization env rebuilt from the
deployment recipe's .env — the recipe is the source of truth for those
values; never scrape them from a live process's /proc and never expand
them through create/exec argv. The exact procedure (env file, mode 600,
sourced inside a supervisor session) is
references/access-and-lifecycle.md § Recovering lost create-time env — run
it after the restart, then verify:
Fallback — sandbox older than the token-safe window (long-running
deployment, or workshop state you can afford to lose): recreate through the
deployment recipe's own machinery (scripts/03-sandbox.sh /
bring-up.sh), which owns sandbox lifecycle and injects the authorization
env its supported way. That recreate boots the STOCK policy template and
wipes the container filesystem, so afterwards re-run Phase 1 (apply the
workshop blocks), this phase's restart (the fresh sandbox is well inside
the token window), and Phases 2b/3 — every step is idempotent. Do not
hand-roll a delete/create in this flow.
Do nothing here in the normal flow. The learner sets NVIDIA_API_KEY in
the workshop's own Secrets Manager tile, which writes the repo-root
secrets.env that every notebook load_dotenv()s. Leaving the variable absent
at server-launch time is deliberate, not an oversight:
start-jupyter.shset -a-sources secrets.env into the Jupyter server env at
launch, and kernels inherit that env. load_dotenv() does not override
variables already present in the environment — so anything baked in at launch
shadows later edits to the file. Stage a key before launch and the learner's
Secrets Manager change is silently ignored until the server restarts; leave it
unset and load_dotenv() stays authoritative, so the key takes effect on the
next cell run with no restart. This is exactly why the Tavily and LangSmith
keys never exhibited the problem — they were never injected.
scripts/verify-sandbox-ready.sh agrees: an absent key is a PASS in its
default mode. Set EXPECT_PRESEEDED=1 only when auditing an image that is
supposed to carry a baked-in key.
⚠️ Never fall back to COMPATIBLE_API_KEY / OPENAI_API_KEY. Those name
the NemoClaw agent's inference credential — in the community example an
sk-… key for the host TLS proxy (NEMOCLAW_ENDPOINT_URL), not a
build.nvidia.com nvapi-… key. The old one-liner did this and produced a
secrets.env that looked correctly populated while every notebook died with
AuthenticationError: 401 against integrate.api.nvidia.com. Real incident;
stage-nvidia-key.sh now refuses that fallback and warns on any non-nvapi-
key.
Only pre-seed a key when you actually want one baked in (unattended classroom
image), and pass a real nvapi-… key explicitly:
printf'%s''nvapi-…' | SANDBOX=<sandbox> bash scripts/stage-nvidia-key.sh
# or, from a dotenv file that carries a genuine NVIDIA_API_KEY:
SANDBOX=<sandbox> bash scripts/stage-nvidia-key.sh --env-file ./.env
The script merges (never truncates) — the Secrets Manager tile rewrites the
same file wholesale, so a truncating write would destroy keys the learner
already set. It reports key NAMES only, never values. If you pre-seed while
JupyterLab is already up, re-run start-jupyter.sh (token/URL survive) so
kernels pick the key up.
Never paste keys through the agent's chat channel (the in-sandbox agent will
itself refuse them). Tavily (modules 1/2/5 search) and LangSmith (module-3
tracing) are handled the same way — set them in the Secrets Manager, or pass
them to the same script; their policy blocks are in references/policy-blocks.md too.
Phase 2b — Stage this example's skills into the agent library
A fresh sandbox has neither the workshop repo nor any workshop skill, so the
in-sandbox agent has nothing to run yet. Stage this example's skill copies
into the agent's skill library — they register as local/enabled on the
agent's next session, and setup.sh's final step deliberately treats staged
copies as canonical (it never overwrites them from the clone):
The script stages transactionally (hidden temp dir, chown scoped to what it
staged, single-exec swap — a failed copy leaves the existing skill intact)
and excludes this host-side skill, which would only mislead the in-sandbox
agent. docker cp/docker exec are fine here — filesystem staging, not an
egress/syscall probe. Minimum viable staging is setup-workshop-nemoclaw
alone (setup then fills the tutor skills from the clone), but staging the
full set keeps the library on this example's adjusted copies.
Phase 3 — Kick the in-sandbox agent
Message the sandbox agent. Without a chat channel wired up, the working path
is a one-shot hermes session under real sandbox enforcement (openshell sandbox exec rejects multi-line args — keep the prompt on one line):
--accept-hooks keeps the one-shot session non-interactive: it pre-accepts
the hook-consent prompt for hooks already configured in the deployed agent
stack. The workshop repo is not yet cloned when this runs, so no workshop
content can ride in on the flag (data-handling summary: example README,
"Security and data-handling considerations").
Egress policy for the Build-an-Agent workshop is applied: the workshop-repo
clone route, PyPI installs, and the NIM/reranking endpoints are open. The
setup-workshop-nemoclaw skill is staged in your skill library
(/sandbox/.hermes-data/skills/). Run it (NOT the workshop repo's
bare-metal setup-workshop): it clones the workshop repo, then runs
preflight, setup, and start-jupyter. When JupyterLab is up, save the token
URL to /sandbox/workshop-url.txt and report back; I'll open the forward.
NVIDIA_API_KEY is deliberately not staged — the learner sets it in the
Secrets Manager tile.
NemoClaw-specific gotcha: the agent's persona (SOUL.md) may still say
GitHub/PyPI/serving are off-limits, making it refuse without trying. The
gateway caches SOUL.md at startup — after editing the durable copy and
uploading to /sandbox/.hermes-data/SOUL.md, either restart the agent stack
or (simpler) tell the live agent explicitly what is now allowed, as above.
Do not relay environment guesses as facts — a wrong relayed TLS path once
cost real round-trips; the in-sandbox skill now carries the correct values.
Phase 4 — Open the access path (per session)
The agent's Jupyter binds the sandbox inner loopback (127.0.0.1:8888).
Agent processes live in an inner network namespace — a server bound even to
0.0.0.0 is unreachable at the container IP. The only inbound path:
# 1. On this host (FOREGROUND — leave the terminal open). Note: `openshell# forward list`/`stop` do NOT track `forward service` tunnels — stop it# with Ctrl-C or pkill -f "forward service $SANDBOX".
openshell forward service "$SANDBOX" --target-port 8888 --local 8888
# Agent-driven runs (e.g. Claude Code): a session-tied background task dies# with the session, silently cutting the user's access (happened live).# Start it detached instead, so it outlives the session:# setsid nohup openshell forward service "$SANDBOX" \# --target-port 8888 --local 8888 >/tmp/forward-"$SANDBOX".log 2>&1 </dev/null &# 2. Sanity check from another host shell (302 = alive, auth redirect):
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8888/lab
# 3. Get the token URL (gateway masks tokens in agent chat, hence the file):
docker exec"$C"cat /sandbox/workshop-url.txt
Then from the laptop, ONE of (full Teleport troubleshooting in
references/access-and-lifecycle.md):
ssh -N -L 8888:localhost:8888 <user>@<sandbox-host> # vanilla ssh
tsh ssh -N -L 8888:localhost:8888 <user>@<node-name> # Teleport: NODE NAME from `tsh ls`, not DNS/IP
-N is silent when it works — no output means the tunnel is UP; open the
browser before assuming a hang. Finally open the token URL:
http://localhost:8888/lab?token=….
Hand-off rule — the final access-steps message ALWAYS includes the
host-side forward command, even when a forward is already up and serving.
The forward is the one leg the user cannot see: forward list does not
track it, and it dies silently with whatever shell or agent session spawned
it (a Claude Code exit cut a user's access exactly this way). Alongside the
laptop tunnel + token URL, hand the user something like:
FYI — the JupyterLab URL depends on a port-forward running on the sandbox
host. If the URL stops responding, restart it there:
openshell forward service <sandbox> --target-port 8888 --local 8888
(leave it running; stop with pkill -f "forward service <sandbox>"), then
reload the page. Re-read the token URL any time with:
docker exec <container> cat /sandbox/workshop-url.txt.
If the deployment's .env configured Slack (or Outlook), append an optional
pointer — the resident agent now carries the workshop tutor skills on every
channel it serves:
Optional: your deployment's Slack bot is the same sandboxed agent — DM it
a workshop question (e.g. "quiz me on module 1") to meet your tutor
outside JupyterLab.
Phase 5 — When something is denied
The L7 proxy/OCSF audit log names the exact process path and rule for every
verdict:
Feed that line back into the policy (right block, right binary — enforcement
resolves symlinks, so e.g. git-remote-https → list git-remote-http too),
re-apply, re-verify. If the in-sandbox agent reports a blocked call, ask it
for the exact URL/error and match it against the log.
Two verdict patterns that are NOT policy gaps (both observed live):
Boot-time noise: a /usr/bin/python3.13 DENIED to github.com:443
("binary not allowed in policy 'github_git_clone'") right after sandbox
start is agent-stack startup traffic. Python is deliberately absent from
that block's binaries — ignore it; it is not workshop breakage.
First-touch flake: a probe reports curl 000 while the audit log shows
ALLOWED at both engines for that same request — the first request to a host
through the L7 proxy can stall past curl's timeout. Retry before editing
policy (verify-sandbox-ready.sh retries its clone-route probe once for
exactly this reason).
Lifecycle pitfalls (each caused real breakage)
docker restart is safe ONLY inside the bootstrap-token window. The
container boots from a static bootstrap JWT (~1-hour TTL, written once at
create to /etc/openshell/auth/sandbox.jwt and never refreshed on disk).
Within the window a restart recovers cleanly and boots dormant
filesystem_policy grants (verified on OpenShell v0.0.96: Ready in ~10 s);
past it the supervisor re-reads the stale token and crash-loops
(ExpiredSignature), sticking the sandbox in Provisioning (verified on
v0.0.53 and v0.0.96) — recovery then needs a recreate. Use the Phase 1b
age guard; never restart an aged sandbox.
Gateway upgrades restart sandbox containers. An upgraded gateway
resumes sandboxes by restarting their containers, so every sandbox older
than its token window bricks exactly as above (observed live during the
0.0.53 → 0.0.96 upgrade). Plan sandbox recreates around gateway upgrades.
A container restart does NOT relaunch the agent stack (nemoclaw-start:
agent, relay, bridges — and JupyterLab). Relaunch the stack per
references/access-and-lifecycle.md § Recovering lost create-time env, then
have the agent re-run start-jupyter.sh.
A sandbox recreate wipes the container filesystem (venv, shim,
secrets.env, the server), and the sanctioned recreate path is the
deployment's own machinery (scripts/bring-up.sh /
scripts/03-sandbox.sh) — it re-renders the STOCK template, so every workshop grant
(network AND filesystem) reverts by design. Afterwards re-run Phase 1, the
Phase 1b restart (fresh sandbox — inside the token window), and Phase 2b,
then have the agent re-run setup.sh + start-jupyter.sh (all
idempotent). Verify the create-time authorization env reached the
relaunched gateway (Phase 1b check) — a Slack bot that answers everyone
with pairing codes plus a missing outlook-bridge.py process means it
didn't; fix per references/access-and-lifecycle.md, no re-recreate needed.
docker exec proves nothing about the agent's sandbox (1 seccomp filter
vs the agent's 4, no Landlock). Egress/syscall tests: openshell sandbox exec only.
The netlink/seccomp kernel-startup block has NO operator knob — it is
compiled into the in-container OpenShell supervisor (Rust seccompiler), not
the Docker seccomp JSON, not the policy schema. The sanctioned fix is the
in-sandbox LD_PRELOAD shim (the sandbox skill builds it). Do not edit Docker
seccomp JSON for this; it has no effect.
Never route workshop execution through docker exec to dodge seccomp —
it also escapes Landlock egress enforcement. Forbidden.
End-to-end verification checklist
openshell policy get "$SANDBOX" shows the new revision; probes via
openshell sandbox exec: pypi 200, integrate.api.nvidia.com/v1/models 200.
Skills staged: docker exec "$C" ls /sandbox/.hermes-data/skills lists
setup-workshop-nemoclaw (+ workshop, module-1…module-7 if the full
set was staged).
(only if a key was pre-seeded) docker exec "$C" ls -l /sandbox/workshop-build-an-agent/secrets.env → present, mode 600.
Terminal tile opens a shell (POST /api/terminals with token → 200) —
needs the /dev/pts grant BOOTED. An auto-hidden tile means the Phase
1b restart was skipped; on a fresh sandbox run it before setup.
(if a Slack/Outlook channel is configured) a DM to the deployment's bot
with a workshop question routes to the tutor skills — surface this
option to the user alongside the JupyterLab URL (Phase 4).
