debug-codex-session

name

debug-codex-session

description

Investigate a specific Codex CLI session by session ID — correlate the local rollout transcript (`~/.codex/sessions/YYYY/MM/DD/rollout-*-<SESSION_ID>.jsonl`) with the router's production logs to understand what Codex rendered vs. what the upstream served on the /v1/responses path. Use when given a session ID and asked "why did X render?" (missing thinking, missing routing marker, wrong model, tool-call weirdness) for a Codex conversation routed through the router.

Debugging a Codex session

Given a Codex session ID, pull the local rollout transcript (what the client saw) and the corresponding production cloud logs (which model/provider served it), then correlate them. The rollout .jsonl is ground truth for what rendered; the cloud logs confirm what the router decided and what the upstream sent; internal/translate/responses*.go + internal/proxy/service.go explain why the wire shape looks that way.

The sibling skill debug-claude-session is the Claude Code counterpart. The workflow is the same shape; the transcript format, the id conventions, and the correlation key are all different — do not carry Claude assumptions over.

Setup: Cloud deployment config

Before starting, create a gitignored config file with your deployment's cloud logging details:

cat > .claude/skills/debug-codex-session/.deployment.json <<'EOF'
{
  "cloud_provider": "gcp",
  "project_id": "your-project-id",
  "region": "us-central1",
  "service_name": "router",
  "log_command_template": "gcloud logging read ... --project {project_id} --format=json"
}
EOF

If .deployment.json is missing, prompt the user for these details and walk them through creating it. The file is gitignored and contains no secrets — just the service/project/region names needed to construct cloud log queries.

Critical gotchas (read first)

• Correlate by client_session_id, not by time. Codex 0.149+ sends the session ID as the Session-Id header (and Thread-Id with the same value on the main thread); sessionIDFromHeaders (internal/proxy/client_identity.go) picks it up and bindRequestLogger binds it as client_session_id on every log line. So the Codex session ID from the transcript filename is a direct log filter. This is the big difference from the Claude skill, which has to correlate by time + model.
• The transcript does NOT record the served model. turn_context.model and thread_settings_applied.model are the model Codex requested (e.g. gpt-5.6-sol). The router ignores the request's model field for routing. The served model appears only in (a) the injected Weave routing marker, when one was emitted, and (b) decision_model in cloud logs. Never report turn_context.model as the model that served the turn.
• Empty summary: [] on a reasoning item is not lost thinking. Native Codex reasoning items carry opaque encrypted_content (replay state for the next Responses request) and a public summary array. With summary: "auto" the upstream frequently returns zero summary text while still billing reasoning_output_tokens. Check event_msg/token_count → info.total_token_usage.reasoning_output_tokens before concluding thinking was dropped — nonzero tokens with empty summaries means the model reasoned and the upstream chose not to surface a summary.
• Codex subscription turns take the native passthrough path — the router does not translate them. codexResponsesRequest (internal/proxy/service.go) detects a ChatGPT JWT + ChatGPT-Account-ID; the original Responses body is preserved and ResponsesWriter.SetPassthrough() / SetPassthroughBadge() (internal/translate/responses.go) forward upstream bytes verbatim. On that path, "the router dropped it" is almost never the answer — verify passthrough before blaming translation.
• ProxyOpenAIChatCompletion in the logs does not mean Codex used chat completions. delegates into the shared chat proxy for routing/billing/telemetry, so the completion log line is named and even for traffic. Check on the access-log line to see the real surface.

Workflow

- [ ] 1. Locate the local rollout transcript
- [ ] 2. Get the session shape (event histogram + session_meta)
- [ ] 3. Read turn_context: requested model, effort, summary mode
- [ ] 4. Extract the items showing the symptom
- [ ] 5. Check token accounting before calling anything "missing"
- [ ] 6. Fetch cloud logs filtered by client_session_id
- [ ] 7. Correlate transcript + cloud logs
- [ ] 8. Trace to the responses code path

1. Locate the local rollout transcript

Codex writes one rollout file per session, date-partitioned, with the session ID as the filename suffix:

find ~/.codex/sessions -name "rollout-*<SESSION_ID>.jsonl" -type f

You get ~/.codex/sessions/<YYYY>/<MM>/<DD>/rollout-<ISO8601>-<SESSION_ID>.jsonl — one JSON object per line, no sibling spillover directory (unlike Claude Code). wc -l it; a working session is typically hundreds of lines.

Note the ISO timestamp in the filename is local time; the timestamp field inside each line is UTC (Z-suffixed). Use the in-file timestamps for any time math.

2. Get the session shape

Every line is {"timestamp", "type", "payload"}. The type is the outer kind; payload.type is the specific event. Histogram first — it tells you at a glance whether the session compacted, aborted, or ran tools:

python3 - <<'EOF'
import json, collections
F = "<path>/rollout-...-<SESSION_ID>.jsonl"
c = collections.Counter()
for line in open(F):
    try:
        o = json.loads(line)
    except Exception:
        continue
    c[(o.get("type"), (o.get("payload") or {}).get("type"))] += 1
for k, v in c.most_common():
    print(v, k)
EOF

The kinds you'll see:

type	payload.type	meaning
session_meta	—	one per file: session_id, cwd, originator, cli_version, source, model_provider, git, context_window
turn_context	—	per-turn settings: model, effort, summary, approval_policy, collaboration_mode
response_item	message	a conversation message (role: developer / user / assistant)
response_item	reasoning	id (rs_*), summary[], encrypted_content
response_item	custom_tool_call / function_call	tool invocations (ctc_* / fc_* ids, call_id)
response_item	custom_tool_call_output / function_call_output	results, keyed by call_id
event_msg	token_count	cumulative + last-turn usage, plus rate_limits
event_msg	agent_message	what the TUI rendered as assistant text — routing markers show up here
event_msg	task_started / task_complete / turn_aborted	turn lifecycle (turn_id, timings, )

model_provider on session_meta tells you whether the session went through the router at all — it's the provider id from ~/.codex/config.toml (e.g. weave). If it isn't the router's provider id, stop: this session never hit the router.

3. Read turn_context

python3 - <<'EOF'
import json
F = "<path>/rollout-...-<SESSION_ID>.jsonl"
for i, line in enumerate(open(F), 1):
    o = json.loads(line)
    if o.get("type") != "turn_context":
        continue
    p = o["payload"]
    print(f"line {i}: turn_id={p.get('turn_id')} requested_model={p.get('model')} "
          f"effort={p.get('effort')} summary={p.get('summary')}")
EOF

effort and summary are the two knobs that decide how much reasoning is produced and how much of it is displayable. summary: "auto" is the usual cause of a session with lots of reasoning tokens and no visible thinking. Again: model here is the request, not the route.

4. Extract the items showing the symptom

Adapt this to your symptom. The template below covers the common "no thinking rendered" case plus assistant-message provenance:

python3 - <<'EOF'
import json
F = "<path>/rollout-...-<SESSION_ID>.jsonl"
reasoning = empty_summary = 0
for i, line in enumerate(open(F), 1):
    o = json.loads(line)
    p = o.get("payload") or {}
    t = p.get("type")
    if t == "reasoning":
        reasoning += 1
        summary = p.get("summary") or []
        enc = p.get("encrypted_content") or ""
        if not summary:
            empty_summary += 1
        if reasoning <= 3:  # sample the first few
            print(f"line {i}: rs id={p.get('id')} summary_parts={len(summary)} "
                  f"encrypted_len={len(enc)}")
    elif t == "message" and p.get("role") == "assistant":
        mid = p.get("id") or ""
        # short base36 tail => router-minted (badge); long hex => upstream
        origin = "router" if len(mid) < 20 else "upstream"
        text = "".join(b.get("text", "") for b in (p.get("content") or []))
        print(f"line {i}: assistant id={mid} origin={origin} {text[:80]!r}")
print(f"\nreasoning items: {reasoning}, with empty summary: {empty_summary}")
EOF

To see exactly what the TUI rendered (including markers), read the agent_message events instead — those are the client-side render records:

python3 - <<'EOF'
import json
F = "<path>/rollout-...-<SESSION_ID>.jsonl"
for i, line in enumerate(open(F), 1):
    o = json.loads(line)
    p = o.get("payload") or {}
    if p.get("type") == "agent_message":
        print(f"line {i}: {(p.get('message') or '')[:120]!r}")
EOF

The routing marker renders with invisible provenance characters (⁣⁠) prefixed by EnableCodexBadgeProvenance — that prefix is how the router recognizes and strips its own badge on replay. Seeing those codepoints confirms the marker came from the router, not the model.

5. Check token accounting before calling anything "missing"

python3 - <<'EOF'
import json
F = "<path>/rollout-...-<SESSION_ID>.jsonl"
last = None
for line in open(F):
    o = json.loads(line)
    p = o.get("payload") or {}
    if p.get("type") == "token_count":
        last = p
if last:
    print(json.dumps(last.get("info", {}).get("total_token_usage", {}), indent=2))
    print("rate_limits:", json.dumps(last.get("rate_limits", {}))[:300])
EOF

reasoning_output_tokens > 0 with zero rendered summaries = the model reasoned and the summary was withheld upstream, not a router bug. output_tokens vs reasoning_output_tokens also tells you whether an apparently empty turn produced anything at all.

6. Fetch cloud logs filtered by client_session_id

Unlike the Claude path, filter directly on the session ID:

gcloud logging read \
  'resource.type="cloud_run_revision"
   AND resource.labels.service_name="router"
   AND jsonPayload.client_session_id="<SESSION_ID>"' \
  --project <project_id> --limit 100 --format=json \
  > /tmp/codex_logs.json

If that returns nothing, widen in this order before assuming the session bypassed the router:

1. Add a UTC time window from the transcript (timestamp of first and last line) — log retention and index lag both bite.
2. Try a free-text search for the id: '"<SESSION_ID>"' — it may appear on lines emitted before bindRequestLogger runs.
3. Check session_meta.model_provider (step 2) — a non-router provider means there is nothing to find.

Then summarize the routing decisions:

python3 - <<'EOF'
import json
for entry in json.load(open("/tmp/codex_logs.json")):
    p = entry.get("jsonPayload", {})
    if not p.get("decision_model"):
        continue
    print(json.dumps({
        "timestamp": entry.get("timestamp"),
        "message": p.get("message"),
        "requested_model": p.get("requested_model"),
        "decision_model": p.get("decision_model"),
        "decision_provider": p.get("decision_provider"),
        "decision_reason": p.get("decision_reason"),
        "upstream_status": p.get("upstream_status"),
        "sticky_hit": p.get("sticky_hit"),
    }))
EOF

ProxyOpenAIChatCompletion complete is the per-turn summary line for Codex traffic (see gotchas — the name is an artifact of ProxyOpenAIResponses delegating into the shared path). Its fields answer most questions on their own: decision_model, decision_provider, decision_reason, failover_used, upstream_status, route_ms, proxy_ms.

7. Correlate transcript + cloud logs

Join on:

• client_session_id (exact — the primary key for Codex).
• turn_id from the transcript ↔ turn boundaries in the logs, by timestamp within the session.
• Served model: cloud decision_model ↔ the model named in the transcript's routing marker. If they disagree, the marker was stale or stripped — trust the log.

What the logs then tell you: which model/provider actually served, whether a fallback or failover fired, whether the pin was sticky, whether the upstream returned a non-200, and how usage was accounted.

8. Trace to the responses code path

Codex traffic enters at internal/api/openai/responses.go → proxy.Service.ProxyOpenAIResponses (internal/proxy/service.go). From there the path forks:

Situation	Path	Code
ChatGPT subscription (JWT + ChatGPT-Account-ID)	native Responses passthrough, bytes forwarded verbatim	codexResponsesRequest, ResponsesWriter.SetPassthrough / SetPassthroughBadge in internal/translate/responses.go
Prepaid / BYOK, OpenAI decision	Responses → chat completions → Responses re-emit	ConvertResponsesToChatCompletionsWithOptions, emit_openai_responses.go
Non-OpenAI decision (Anthropic/Gemini/compat)	Responses → chat → provider format → Responses re-emit	responses_to_openai_chat_writer.go, emit_openai_responses_from_openai.go
Badge / marker injection + strip	synthetic assistant item, provenance-prefixed	emitNativeBadgeBeforeOutput, StripRoutingBadgeFromResponsesInput in responses.go
Codex-specific item shapes	custom_tool_call vs function_call id prefixes	responses_codex.go, toolCallItemIDPrefix

Read backward from the emitter to the upstream event that triggered it. On the passthrough path there is no emitter — the bytes are the upstream's, so the answer lives upstream or in the request Codex built.

Example: "zero thinking is showing up"

1. Transcript located at ~/.codex/sessions/2026/08/28/rollout-2026-08-28T09-10-06-<id>.jsonl (step 1).
2. Histogram: 101 reasoning items, 99 token_count events, 93 custom_tool_call pairs (step 2).
3. turn_context: requested gpt-5.6-sol, effort: "high", summary: "auto" (step 3).
4. Every reasoning item has summary: [] and 1,400–4,600 chars of encrypted_content (step 4).
5. Final token_count: ~8,300 reasoning_output_tokens — the model definitely reasoned (step 5).
6. Cloud logs filtered by client_session_id show decision_model=gpt-5.6-luna, decision_provider=openai, HTTP 200s, no upstream errors (step 6).
7. Path is native Codex passthrough: SetPassthrough() forwards upstream bytes unchanged (step 8).
8. Verdict: not an inference bug (reasoning tokens produced), not a translation-loss bug (encrypted reasoning preserved verbatim). The upstream returned encrypted reasoning with an empty public summary under summary: "auto", so Codex has no text to render. Synthesizing text from encrypted_content is impossible and would be wrong.

Notes

• On the passthrough path the transcript records the native Responses shape — the same bytes the upstream sent, minus what Codex chose not to persist. On the translated path it records the router's re-emitted Responses shape, which is where translation artifacts live.
• compacted + context_compacted mark a mid-session context handover; items before it may not reflect what the model actually saw afterward. Check for them before reasoning about "the model ignored X".
• turn_aborted with reason: "interrupted" is a user cancel, not a failure — don't chase it in the logs.
• To reproduce a Codex-path artifact locally (without prod), use the sibling test-codex-locally skill.