sandboxed-adversarial-test-harness

name	sandboxed-adversarial-test-harness
description	Design and audit adversarial test harnesses that prove a sandbox contains untrusted or AI-authored code and agent actions — an active adversary trying to break isolation, exfiltrate secrets, or cause side effects, not just the code's own passing unit tests. Use when gating a sandbox, output sink, worktree, or trust gate (e.g. ADR-0093's event-spawn trust substrate) before it holds real agent actions, when defining containment invariants for filesystem/network/secrets/process isolation, or when deciding fail-closed vs fail-open semantics. NOT for training or fine-tuning agents inside a sandbox (use agent-rl-sandbox-trainer), building the trust-gate/allowlist logic itself (use fleet-event-spawn-trust), general zero-trust crypto/capability architecture (use agentic-zero-trust-security), or macOS host-level ESF/Network-Extension implementation detail (use macos-host-security).
license	Apache-2.0
allowed-tools	Read,Write,Edit,Bash,Grep,Glob
metadata	{"category":"Infrastructure & DevOps","tags":["sandboxing","adversarial-testing","containment","ssrf","path-traversal"],"provenance":{"kind":"first-party","owners":["port-daddy"]},"pairs-with":[{"skill":"agent-rl-sandbox-trainer","reason":"Agents should only be trained inside a sandbox this harness has already proven contains them."},{"skill":"runtime-verification-for-agents","reason":"Runtime monitors watch live sessions for the same invariants this harness tests statically before deploy."},{"skill":"agentic-zero-trust-security","reason":"Supplies capability/ocap and signed-envelope vocabulary the harness's threat model builds on."},{"skill":"whitehat-defense","reason":"Shares the red-team-round / shore-up reporting format this harness's adversarial log uses."},{"skill":"macos-host-security","reason":"Supplies concrete sandbox-exec, ESF, and Network Extension mechanisms cited in the isolation-mechanisms reference."}],"io-contract":{"kind":"deliverable","consumes":["[Truncated]","[Truncated]"],"produces":["[Truncated]","[Truncated]","[Truncated]"]}}

Sandboxed Adversarial Test Harness

Prove a sandbox contains untrusted or AI-authored code under active attack — not just that the code's own tests pass.

Use This For

• Gating a fleet output sink, worktree, or trust gate (ADR-0093's event→spawn trust substrate) before it holds real agent-triggered actions.
• Verifying the confirmed SSRF and path-traversal fixes on lib/fleet/url-guard.ts / lib/fleet/path-guard.ts actually hold against obfuscated payloads, not just the one exploit that was found.
• Defining containment invariants for a new sandbox (filesystem, network egress, secrets, process/resource limits) before it runs its first untrusted payload.
• Deciding whether a sandbox's failure semantics are fail-closed or fail-open, and refusing to ship a fail-open design.
• Unblocking Phase 4E/4F hardening on the V4 roadmap, which is stalled behind exactly this harness existing.

Do Not Use This For

• Training, fine-tuning, or QLoRA-adapting an agent's behavior — that is agent-rl-sandbox-trainer; this skill only proves the box the training happens in is sound.
• Writing the trust-gate classification or allowlist logic itself — that is fleet-event-spawn-trust; this skill tests whether that logic holds under attack.
• Reviewing the code's own correctness or business logic — a sandbox can contain buggy code perfectly well; containment and correctness are orthogonal.

Containment-Proving Process

flowchart TD
  A[Define isolation boundary] --> B[Enumerate adversary goals per threat class]
  B --> C[Write one adversarial case per threat class in scope]
  C --> D[Decide fail-closed vs fail-open per case]
  D --> E[Run scripts/containment_audit.mjs]
  E --> F{pass and zero findings?}
  F -->|No| C
  F -->|Yes| G[Gate deploy on green harness]
  G --> H[Name residuals honestly; re-audit on guard code changes]

1. Define the isolation boundary explicitly: filesystem jail root, network egress policy, what secrets (if any) the sandboxed code can reach, process/resource limits. Name each dimension; do not leave any implicit.
2. Enumerate what an adversary inside the sandbox actually wants, per threat class: ssrf (reach metadata/loopback/internal hosts), path-traversal (escape the jail root), secret-exfil (move a credential out via any channel), resource-exhaustion (fork bombs, memory/disk bombs), side-effect-write (write outside the intended output surface, e.g. cron, launch agents, git hooks).
3. Write at least one adversarial case per in-scope threat class. Each case names the invariant it asserts, not the exploit's mechanics — "outbound fetch to 169.254.169.254 is refused pre-socket," not "test SSRF."