| name | manual-e2e-qa |
| description | Manual end-to-end QA playbook for decode against a real provider — what to type at each surface (chat, gated read/write, bash, todo, web_fetch, ask_user, lsp, agent subagents, docker/modal sandboxing, headless `decode run`, HITL, model override, replay, mid-turn steer/abort, persistence/memory, /ship, sandbox git token) and what "working" looks like. Use when manually testing or verifying a decode surface end-to-end. |
Testing E2E
Automated M1 proof: capstone tests/integration/test_milestone1_capstone.py — six-step conversation (read → gated write approve → gated write deny → todo_write → ask_user → web_fetch) through real build_agent() + Runner + render_event + session log + memory write-back; only network boundary swapped (FunctionModel for model, httpx.MockTransport for web tool). Run: make integration-tests or make ci. No API key, no network.
Below: manual e2e vs real Gemini — exercise each surface, then try to break it (adversarial half = Tester's job).
Launch (one env var, no service):
export GEMINI_API_KEY=…
uv run decode
No GEMINI_API_KEY → one friendly stderr line — decode: set GEMINI_API_KEY in your environment or .env to start (see .env.example). — exit non-zero, no traceback (task-004 guard in cli.py).
Per surface: what to type + what "working" looks like.
| Surface | Type this | Working looks like |
|---|
| Plain chat | what can you do? | answer streams token-by-token above prompt; prompt pinned at bottom. |
| Read (gated) | read pyproject.toml | permission? read … prompt; y → panel with numbered file contents; n → model told denied, adapts. |
| Write (gated, approve) | create a file hello.txt that says hi | permission? write … → y → file appears on disk (cat hello.txt), model confirms. |
| Write (gated, deny) | repeat write, answer n | file not created (ls hello.txt → absent); model told denied (doesn't pretend it wrote). |
| Bash | run the tests with make unit-tests | permission? bash … → y → panel with stdout/stderr (truncated past cap); runaway command bounded by bash_timeout_s. |
| Todo checklist | make a 3-step plan to add a CLI flag and track it | blue tasks panel renders checklist ([ ] / [~] / [x]), re-renders as statuses update. |
| web_fetch | fetch https://example.com and summarize it | permission? web_fetch … → y → page as Markdown (HTML stripped), model summarizes. |
| ask_user | deploy my app (underspecified) | model calls ask_user; ask: … question + type your answer: cue; next typed line is the answer, turn resumes. |
| lsp (code intelligence) | where is build_agent defined? |
Sandboxing (ADR-0012 — see invariants above for the seam / Workspace / fresh-exec / Harness Home model). Default SANDBOX_MODE=none keeps every row above byte-unchanged. In docker / modal the whole tool scope — file/search tools and bash — is the one Workspace; decode's artifacts stay at Harness Home. --repo/SANDBOX_REPO needs a sandbox mode; with SANDBOX_MODE=none → Decode: --repo/SANDBOX_REPO clones a repo into the isolated sandbox Workspace, which only exists in a sandbox mode …, non-zero, no traceback.
| Surface | Type this | Working looks like |
|---|
| file tools + bash (docker Sandbox) | launch SANDBOX_MODE=docker decode --repo <url-or-local-path> (needs running Docker daemon; --local = fast local-path clone), then glob **/*.py, read one of them, write a NOTES.md, then run 'echo hi > side.txt && cat side.txt' | launch: Decode - cloning <repo> into the workspace… (only when repo given AND Workspace still empty), then Decode - starting docker sandbox (ghcr.io/astral-sh/uv:python3.12-bookworm-slim)… + sandbox:docker banner segment — docker ps shows a sleep infinity container before any bash (clone failure degrades to empty Workspace + one friendly line). File/search tools + bash share the one /workspace tree — live bind mount, always truthful: bash-written file → read; bash rm → glob. bash still gates; fresh-exec (chain: cd /workspace/app && …); timeout kills only that docker exec — container + filesystem survive, reply says timed out. lsp + post-edit diagnostics run (host ty reads bind mount). git installed into container at startup (slim base ships none), identity preconfigured (SANDBOX_GIT_USER_*, default decode/decode@localhost) → model git commit in /workspace works; push/PR creds stay host-side (hand-back) unless you opt into SANDBOX_GIT_TOKEN (own row). web_fetch stays gated (host network). Exit or /ship → hand-back (see /ship row). QA peek: docker exec -it <id> bash (filter ancestor=ghcr.io/astral-sh/uv:python3.12-bookworm-slim) → /workspace = host .decode/sandbox. Guard: daemon stopped → Decode: SANDBOX_MODE=docker but the Docker daemon is not reachable — start Docker and retry (see .env.example)., non-zero, no traceback. |
Mid-turn interaction (while a turn streams — ADR-0002 §4-5):
- Steer — type a line + plain Enter: injected at next model-request boundary (never mid-stream/mid-tool).
- Follow-up — Alt+Enter: queued, drained only when turn would otherwise stop; continues as new turn.
- Abort — Esc: turn stops at next boundary, keeps work done, REPL idle (
[aborted] marker).
Persistence + memory across sessions:
decode --resume (or decode --resume <session-id>) replays latest (or named) session log from .decode/sessions/*.jsonl; prior conversation seeded, continue it.- On quit (
/quit or Ctrl-D), one cheap Gemini call appends a dated one-line summary (- YYYY-MM-DD: …) to .decode/MEMORY.md; relaunch injects it into agent instructions.
