test-codex-locally

name

test-codex-locally

description

Run the Weave router locally in docker compose and drive it with `codex exec` to reproduce and verify routing/translation/marker behavior for Codex's Responses API path. Use when verifying a router fix end-to-end for Codex, reproducing a prod Codex routing bug, confirming routing-marker / force-model / subscription-passthrough behavior, or testing a `/force-model` route — without touching the user's global Codex config.

Testing the router locally with Codex

For an automated pre-merge regression net (fixture-driven, asserts caching/streaming/decision-headers against real Anthropic), run make smoke — see docs/SMOKE.md. This skill is the interactive counterpart for the Codex CLI (codex exec): stand the stack up by hand and drive it via a throwaway CODEX_HOME so ~/.codex/config.toml is never edited.

Stand up the router in docker compose, point a one-off codex exec at it via CODEX_HOME, and read the local server logs to confirm behavior. Two upstream modes: the real provider API (needs a working key + credits, and for native GPT models a ChatGPT OAuth login) or a mock upstream that emits an exact SSE shape (deterministic, no credits).

The sibling skill test-claude-locally is the Claude Code (claude -p) counterpart. Stack bring-up and seeding are identical; only the client driver differs.

Critical gotchas (read first)

• Never edit ~/.codex/config.toml or ~/.codex/auth.json. Those are the user's live Codex install (prod Weave Router + ChatGPT OAuth). Redirect with a throwaway CODEX_HOME directory that contains its own config.toml (and a copy of auth.json when ChatGPT OAuth is required). CODEX_HOME is how Codex finds config; there is no --settings flag equivalent to Claude Code.
• requires_openai_auth = true is required. Codex 0.149 hangs or never issues a /v1/responses request if the custom provider uses requires_openai_auth = false or env_key. Copy ~/.codex/auth.json into the throwaway CODEX_HOME so Codex can attach the ChatGPT JWT + ChatGPT-Account-ID. The mock-router experiment that dropped OAuth never even hit the mock.
• Codex always sends a ChatGPT JWT + ChatGPT-Account-ID. The local router will classify that as a Codex subscription (codexResponsesRequest → native Responses passthrough for OpenAI decisions). That is the path this skill is usually exercising (routing markers on verbatim GPT frames). To force the prepaid/BYOK translation path instead, flip subscription_routing_disabled=true on the seeded installation (see step 3).
• codex exec is one-shot. A standalone /force-model call does not persist to the next codex exec (new session id every time, unless you codex exec resume). Put /force-model <model> as the first line of the SAME prompt. The leading space is load-bearing — Codex consumes unknown slash tokens as local commands; a leading space makes it a normal user message the router can parse.
• Pin via config.toml, not codex -c. On Codex 0.149.1, -c 'model_providers.weave-local.http_headers."x-weave-force-model"="…"' did not merge into the outgoing request (no x-weave-force-model applied in server logs; the scorer served something else). Put "x-weave-force-model" = "<id>" on the same http_headers table in the throwaway config.toml (step 4). Confirm with docker compose logs server | grep 'x-weave-force-model applied' before drawing conclusions.
• The router ignores the request's model field for routing (cluster scorer / pins win). Codex's -m gpt-5.5 only changes what Codex requests; /force-model or x-weave-force-model is the only way to pin a specific routed model. Native GPT family (gpt-5.6-sol / terra / luna, plus whatever -m Codex defaulted to) still goes through the scorer unless forced.
• No GNU timeout on macOS. Drive codex exec directly. If a run hangs (Reading additional input from stdin... with no banner), Codex is waiting on a TTY/stdin — do not pipe into it, and make sure -s read-only (or --dangerously-bypass-approvals-and-sandbox) is set.
• Docker may not be running. docker compose fails with a unix://…/.docker/run/docker.sock connect error if Desktop is stopped. open -a Docker and wait for docker info before step 1.
• Other agents in the same workspace can docker compose down and delete /tmp scratch. If health suddenly 404s or CODEX_HOME vanishes mid-run, re-up the stack and reseed — do not reuse a key you can no longer /validate.
• Session key is apiKeyID + first user message. Reusing the same curl/prompt body on the same key reuses the pin slot. routingMarkerFor then returns empty when PriorServedModel == ServedIdentity() (sticky same-model). For first-turn marker tests, seed a new key or change the first user text. Isolated /validate first: curl -sS -o /dev/null -w '%{http_code}\n' -H "X-Weave-Router-Key: $KEY" http://localhost:8080/validate must be 200.
• Native GPT passthrough + tool-only emits a synthetic badge item. SetPassthroughBadge rewrites existing response.output_text.* events and, when a ChatGPT-subscription turn has no assistant text, inserts a leading assistant message item before native tool/reasoning output. Verify the badge in the Responses SSE/JSON; the Codex TUI can bury the first-turn line under tool chatter.
• GET /v1/models 501 from a mock is fine. Codex probes <base_url>/models first, logs an HTML 501, then continues to POST /v1/responses. The real local router implements GET /v1/models as Anthropic passthrough, so this only shows up against a Python mock.
• Port 8085 conflict. The monorepo's pubsub emulator may already own host port 8085. Drop the router's host binding with a docker-compose.override.yml (see workflow). The server still reaches the emulator over the compose network.
• No credits / no key = no reproduction. If the real upstream returns an error (e.g. OpenRouter "Insufficient credits"), use the mock-upstream path instead.

Workflow

- [ ] 1. Bring up the stack (handle port 8085)
- [ ] 2. Seed an API key
- [ ] 3. Choose upstream + subscription vs prepaid path
- [ ] 4. Write a throwaway CODEX_HOME (copy auth.json)
- [ ] 5. Drive with `CODEX_HOME=... codex exec`, forcing the target model
- [ ] 6. Read local logs to verify behavior
- [ ] 7. Clean up

1. Bring up the stack

cd <router-repo>
# Drop the pubsub host-port binding to avoid an 8085 conflict:
cat > docker-compose.override.yml <<'EOF'
services:
  pubsub-emulator:
    ports: !reset []
EOF
docker compose up -d --build server   # --build picks up code changes
until curl -sf http://localhost:8080/health >/dev/null; do sleep 2; done

The override file is gitignored-by-intent scaffolding — delete it in cleanup.

2. Seed an API key

docker compose run --rm seed

Copy the rk_... key it prints — only the line under Weave Router key (shown once — store it now):. Seed stdout repeats the same token in curl examples and installer snippets; grep -oE 'rk_[A-Za-z0-9]+' can grab a stale key from an earlier seed in the same log. Always /validate before driving Codex:

curl -sS -o /dev/null -w '%{http_code}\n' \
  -H "X-Weave-Router-Key: rk_REPLACE_ME" http://localhost:8080/validate
# expect 200

A 401 mid-session usually means the stack was torn down and reseeded (new installation, old token dead).

3. Choose the upstream

Real provider — set the provider key in .env.local (e.g. OPENAI_API_KEY=..., FIREWORKS_API_KEY=...) and restart docker compose up -d server. Confirm the boot log shows <Provider> provider enabled with the real base_url. Use this to confirm a model genuinely produces the behavior.

For an actual Codex subscription response, do not set ROUTER_CODEX_BASE_URL and do not run either mock. Remove that environment entry from docker-compose.override.yml, then rebuild the server:

unset ROUTER_CODEX_BASE_URL  # also remove it from docker-compose.override.yml
docker compose up -d --build server

Use a throwaway CODEX_HOME containing a copy of the real ~/.codex/auth.json, with requires_openai_auth = true and X-App = "codex". The router detects the OAuth bearer plus ChatGPT-Account-ID and calls the real https://chatgpt.com/backend-api/codex/responses endpoint. This consumes the user's ChatGPT quota and requires a valid login; the mock is only for repeatable tests when that endpoint is unavailable or too expensive.

Mock upstream — for a deterministic, credit-free repro of a precise SSE shape. Point the provider's base URL at a local mock and restart. Codex talks to the router (POST /v1/responses); the mock sits behind the router as the upstream the router dispatches to:

python3 .claude/skills/test-claude-locally/scripts/mock_openai_upstream.py >/tmp/mock.log 2>&1 &   # serves :8099
# In docker-compose.override.yml under `server:`, add:
#   environment:
#     FIREWORKS_BASE_URL: http://host.docker.internal:8099/v1
#     FIREWORKS_API_KEY: sk-mock
#   extra_hosts: ["host.docker.internal:host-gateway"]
docker compose up -d server

To exercise the production-equivalent native Codex subscription path, use a native Responses mock instead of the Chat Completions mock:

python3 .claude/skills/test-codex-locally/scripts/mock_codex_upstream.py >/tmp/mock-codex.log 2>&1 &

Add this to the server service in docker-compose.override.yml:

environment:
  ROUTER_CODEX_BASE_URL: http://host.docker.internal:8099/v1
extra_hosts: ["host.docker.internal:host-gateway"]

This preserves real Codex subscription detection (OAuth bearer plus ChatGPT-Account-ID) and native Responses passthrough, while replacing only the outbound ChatGPT backend with a deterministic tool-only stream. The mock emits a function call, so the expected result is a synthetic marker message at output_index: 0, followed by the native tool at output_index: 1.

Edit the mock's emitted chunks to match the upstream shape you're reproducing. Provider→env-var names live in internal/providers/provider.go; base-URL overrides are read in cmd/router/main.go (<PROVIDER>_BASE_URL). For local testing, ROUTER_CODEX_BASE_URL overrides only the subscription backend; the default remains https://chatgpt.com/backend-api/codex.

Subscription vs prepaid (important for marker / passthrough tests):

Codex will attach a ChatGPT JWT. The local router then treats the turn as a Codex subscription:

• OpenAI-family decision → verbatim Responses passthrough (SetPassthrough / SetPassthroughBadge) to the Codex backend (or ROUTER_CODEX_BASE_URL in the native mock setup). This is the path the routing-marker-on-Codex work exercises.
• Non-OpenAI decision → Chat Completions translation + ResponsesWriter.SetBadgeText.

To force the prepaid/BYOK translation path (no ChatGPT backend, uses OPENAI_API_KEY / other provider keys):

docker compose exec postgres psql -U router -d router -c \
  "SET search_path TO router; UPDATE model_router_installations SET subscription_routing_disabled=true WHERE external_id='__router_admin__';"

Revert with subscription_routing_disabled=false when done.

4. Throwaway CODEX_HOME

Do not point this at ~/.codex. A unique temp dir keeps the user's prod Weave + ChatGPT login intact.

export CODEX_HOME=/tmp/weave-codex-local
rm -rf "$CODEX_HOME"
mkdir -p "$CODEX_HOME"
cp ~/.codex/auth.json "$CODEX_HOME/auth.json"   # required; see gotchas
cat > "$CODEX_HOME/config.toml" <<EOF
model_provider = "weave-local"

[model_providers.weave-local]
name = "Weave Router (local)"
base_url = "http://localhost:8080/v1"
wire_api = "responses"
requires_openai_auth = true
http_headers = { "X-Weave-Router-Key" = "rk_REPLACE_ME", "X-App" = "codex" }
EOF

Optional headers on the same http_headers table:

Header	Purpose
"X-Weave-User-Email" = "dev@localhost"	Attribution in local logs
"x-weave-force-model" = "z-ai/glm-5.1"	Preferred pin. Put it here — do not rely on codex -c (see gotchas).
"X-Weave-Routing-Marker" = "off"	Suppress the in-band routing marker

Do not set X-Weave-Router-Strategy unless you are deliberately pinning a strategy — omitting it uses the local deployment default (same rationale as install/install.sh --codex --local).

5. Drive it

cd <scratch-dir-with-files-to-act-on>
CODEX_HOME=/tmp/weave-codex-local \
  codex exec --skip-git-repo-check -s read-only -C "$(pwd)" \
  ' First send exactly: /force-model z-ai/glm-5.1
Then <task that requires tool use>, then stop.'

Notes:

• --skip-git-repo-check lets you run against /tmp or any non-git scratch dir.
• -s read-only (or --dangerously-bypass-approvals-and-sandbox in a throwaway dir) avoids the interactive approval TUI, which codex exec otherwise cannot answer.
• Leading space before /force-model is required (Codex otherwise treats it as an unknown local slash command and never sends it to the router).
• codex exec prints provider: weave-local in its session banner when the throwaway config took; if it prints provider: weave it is still reading ~/.codex — CODEX_HOME was not exported.
• Confirm Codex actually reached the local router: docker compose logs server --since=1m should show a POST /v1/responses (and ProxyOpenAIChatCompletion start).

Headless pin, no slash command — bake the header into the throwaway config.toml (step 4), then:

CODEX_HOME=/tmp/weave-codex-local \
  codex exec --skip-git-repo-check -s read-only -C "$(pwd)" \
  "say hi and stop"

Do not use codex exec -c 'model_providers.weave-local.http_headers."x-weave-force-model"=…' as the pin. On 0.149.1 that override is silently dropped. After the run, grep 'x-weave-force-model applied' in server logs must fire; if it doesn't, Codex never sent the header.

6. Verify via logs

Codex ingress is ProxyOpenAIResponses → ProxyOpenAIChatCompletion. The decision + completion log per action is ProxyOpenAIChatCompletion complete. Strip ANSI first:

docker compose logs server --since=3m 2>&1 | sed -E 's/\x1b\[[0-9;]*m//g' \
  | grep 'ProxyOpenAIChatCompletion complete' | grep 'decision_model=z-ai/glm-5.1'

Useful fields: decision_model, decision_provider, decision_reason, routing_marker, hard_pinned, cross_format, turn_type, proxy_err, upstream_status. Also grep for recovery nudge, tool-call loop, no-progress. SetBadgeText is not logged — look at the Codex transcript / codex output for the ✦ **Weave Router** → <model> (or **Weave Router** — <model> legacy) line.

Confirm the model was actually served before drawing conclusions:

docker compose logs server --since=3m 2>&1 | sed -E 's/\x1b\[[0-9;]*m//g' \
  | grep 'ProxyOpenAIChatCompletion complete' | grep -oE 'decision_model=[^ ]+' | sort | uniq -c

If you only see other models, the pin didn't take. Recheck: (1) x-weave-force-model is in CODEX_HOME/config.toml (not only a -c flag), (2) server log has x-weave-force-model applied, (3) leading space on in-prompt /force-model if you used that path. Codex's session banner model: gpt-5.6-sol is what Codex requested, not what the router served.

Routing-marker specific checks (the usual reason to use this skill):

• Cross-format (non-OpenAI decision, or prepaid OpenAI): first assistant text is ✦ **Weave Router** → <model> · <reason>. Log field routing_marker is non-empty.
• Verbatim GPT passthrough (Codex subscription + OpenAI decision): same marker is injected into the first native assistant text, or as a synthetic assistant message item before tool/reasoning-only output (SetPassthroughBadge). Log still has routing_marker.
• X-Weave-Routing-Marker: off (and not subscription-only warning): no badge. Subscription-only depleted-credits warning still wins over the opt-out.
• Same model, second action (prior_served_model equals decision_model): routing_marker="". Expected. Seed a new key (or change the first user text) to re-see a first-turn badge.
• Tool-call-only on the translated path: marker is a leading output_item (output_index 0) ahead of the function call — including stream:false JSON. Confirm in the SSE / JSON, not only in codex exec stdout (the TUI can bury the first-turn line under tool chatter).
• Tool-call-only on verbatim GPT passthrough: a leading synthetic message item carries the badge, followed by the native tool call. Sequence/output indices are shifted to remain valid.
• Non-Codex client (X-App unset) on a translated decision still gets the ✦ **Weave Router** → … text badge today; native OpenAI passthrough without X-App: codex stays byte-identical (SetPassthrough with no badge rewrite).
• For curl-level checks (no Codex CLI), POST /v1/responses with X-Weave-Router-Key + X-App: codex is enough. Isolate first-turn cases with a freshly seeded key.

7. Clean up

pkill -f mock_openai_upstream.py 2>/dev/null
pkill -f mock_codex_upstream.py 2>/dev/null
rm -rf /tmp/weave-codex-local
rm -f docker-compose.override.yml
# `docker compose down` if you want to stop the stack
# do NOT touch ~/.codex

Notes

• Local cluster version comes from ROUTER_CLUSTER_VERSION in .env.local; it may differ from prod, which is why /force-model (not the scorer) is the reliable way to hit one model.
• Native Codex models gpt-5.6-sol, gpt-5.6-terra, and gpt-5.6-luna use the caller's ChatGPT OAuth plan when subscription routing is on. Other OpenAI / Anthropic / Gemini / OpenAI-compatible models use WorkWeave deployment or BYOK credentials, same as Claude Code.
• To confirm a deploy contains a given router commit: the prod Cloud Run revision name maps to a monorepo commit; git ls-tree <monorepo-commit> router-internal/router shows the pinned router submodule SHA.
• codex exec --ignore-user-config is not a substitute for CODEX_HOME: it skips $CODEX_HOME/config.toml entirely (so you'd have no weave-local provider) while auth still uses CODEX_HOME. Always write a full throwaway config.
• Codex rollout transcripts live under $CODEX_HOME/sessions/<yyyy>/<mm>/<dd>/rollout-*.jsonl. Grep those for Weave Router when codex exec stdout is noisy.
• docker-compose.override.yml is gitignored. If you add a mock *_BASE_URL there, remember to rewrite the file back to the pubsub-only ports: !reset [] (or delete it) before a real-provider Codex run — a leftover mock URL will serve sk-mock against production-looking model names.