challenge-troubleshoot

name	challenge-troubleshoot
description	Use when something in the Simulation Challenge pipeline is misbehaving — auth errors, agent disconnects, jobs stuck in Pending, jobs ending in Failed, drain frames. Maps a symptom to its likely cause and the next command to run.

challenge-troubleshoot — Symptom → cause → next command

When the user reports a problem, do this in order:

1. Reproduce / verify the symptom with a read-only call (/jobs, /result, /log, /current-user-info) before doing anything destructive. For "what's the status" questions, /jobs is the only endpoint that carries one.
2. Match the symptom to the table below.
3. Hand the user the next command from the Action column. Do NOT auto-resubmit jobs (quota cost) or auto-kill agents (loses in-flight cases) without explicit confirmation.

Diagnostic heuristic: 4xx is never a network blip

A structured {"status":"error", ...} body with HTTP 4xx (400/401/403/404) is a semantic rejection by the platform — the request reached the server and was refused on its merits. Retrying it without changing the request gets the same answer and, for POST /api/challenge/job, burns a daily submission slot each time.

Only these qualify as transient and may be retried with backoff:

• TCP connection reset / timeout / DNS failure (curl exits non-zero before getting a response)
• HTTP 5xx (server-side, signals a transient backend failure)

If the user says "可能是网络抖动 / maybe a network blip" but you have a 4xx body in hand, push back: name the actual error and point at the table row.

Diagnostic heuristic: 5xx is often a missing / wrong token, not a real platform outage

curl -fsS collapses any HTTP 5xx to a terse (22) The requested URL returned error: 5XX and hides the body. Before concluding the platform is down, check this in order:

1. State file sourced? Each Bash call should start with [ -f ~/.simubotix-challenge.env ] && . ~/.simubotix-challenge.env — AI assistants spawn every command in a new subshell, so plain export from a previous call is gone.
2. Token presence after sourcing. echo "len=${#CHALLENGE_TOKEN}" — if it's 0, either the file doesn't exist (re-run challenge-login Step 1) or the file exists but doesn't contain the key (the previous login silently failed — see challenge-login Step 1's status check).
3. Token shape. Should be a 3-segment JWT (two dots). echo "$CHALLENGE_TOKEN" | head -c 20; echo — gibberish or null means a previous step persisted a bad value.
4. Body of the actual response. Drop -f so curl prints the body even on non-2xx:

   [ -f ~/.simubotix-challenge.env ] && . ~/.simubotix-challenge.env
   curl -sS -i "$BASE_URL/api/challenge/current-user-info" \
     -H "Authorization: Bearer $CHALLENGE_TOKEN" | tail -15
   

   The body usually says exactly what's wrong (often a 401-style message returned with a 500 status code).

Only after the token is confirmed valid (e.g. the same value just succeeded against /login) should you treat the 5xx as a real platform issue and retry with backoff.

Symptom table

#	Symptom	Likely cause	Action
1	POST /api/challenge/job returns HTTP 400 invalid board / no task templates for board	config.board is missing or not in the currently-supported set	Today the supported values are instruction (default), spatial, manip, robust. Use one of those and resubmit. Do not retry the same wrong board — 400 is a semantic rejection, retrying burns a daily submission slot each time. This holds even when the user says "maybe a network blip" — see the heuristic above. If a contestant insists a new board exists, verify with the organizers first; challenge-submit-job Step 2 is the only list to trust.
2	POST /api/challenge/job returns an "upload limit" error / /api/challenge/submission/quota returns remaining: 0	Today's submission quota is used up	Wait until Beijing midnight (UTC+8), or work with an existing job. Report the actual used / limit from the endpoint — don't quote a cap from memory.
3	Any plain HTTP /api/challenge/* call (login, result, log, job, tunnel-endpoint) returns 401	Invalid / expired CHALLENGE_TOKEN	challenge-login Step 3 to refresh; if refresh also 401s, fall back to Step 1 (email + password). This only affects the plain HTTP APIs — it does not touch a JOB_TOKEN-based tunnel dial.
3b	The WS handshake (GET /api/challenge/tunnel) returns 401	When dialing with JOB_TOKEN (the default per challenge-run-agent), the handshake does not 401 on token expiry — job_tokens don't expire. A 401 here means the ?job= UUID is not owned by this account, the job is in a terminal state, or the ?job= doesn't match the job the token was issued for. If still dialing with CHALLENGE_TOKEN (no JOB_TOKEN available, e.g. an older platform build), a can additionally mean the login token itself expired

Diagnostics shortlist

Run these in order when triaging an unclear failure:

# 1. Token still valid?
curl -fsS "$BASE_URL/api/challenge/current-user-info" \
  -H "Authorization: Bearer $CHALLENGE_TOKEN" | jq

# 2. Job exists, what status? (status lives on the LIST — /result has no status field)
curl -fsS "$BASE_URL/api/challenge/jobs?page=1&per-page=100" \
  -H "Authorization: Bearer $CHALLENGE_TOKEN" \
  | jq -c --argjson id "$JOB_ID" '.items[] | select(.id == $id)
          | {id, status, detailed_status, progress, score}'

# 3. If detailed_status == failed, get the log (read failure_reason first)
curl -fsS "$BASE_URL/api/challenge/job/$JOB_ID/log" \
  -H "Authorization: Bearer $CHALLENGE_TOKEN" | jq

# 4. Scores, once terminal
curl -fsS "$BASE_URL/api/challenge/job/$JOB_ID/result" \
  -H "Authorization: Bearer $CHALLENGE_TOKEN" | jq

# 5. Gateway endpoint sane?
curl -fsS "$BASE_URL/api/challenge/tunnel/endpoint" \
  -H "Authorization: Bearer $CHALLENGE_TOKEN"

$JOB_ID must be the numeric id; the UUID belongs only to the tunnel dial. Passing a UUID to these paths yields 400 invalid job_id, which curl -fsS hides behind a bare exit code 22.

Things NOT to do

• The website/API domain is robocoliseum.ai, but the gateway host is fixed at 120.92.88.78 and is a separate host: BASE_URL=https://robocoliseum.ai, TUNNEL_ENDPOINT=ws://120.92.88.78/api/challenge/tunnel. Do not "fix" the tunnel URL by swapping in the website domain. Prefer a tunnel_endpoint from the job response if present, but the fixed default is a safe fallback.
• Do not reconnect after a drain — the platform is finishing up, and the connection will be rejected.
• Do not spam POST /api/challenge/job to "retry" — each call consumes one of the day's submission slots. Remaining quota is not authorization to guess (e.g. guessing a board because "quota 还有"). Skill rules forbid the guess itself, independent of remaining slots.
• Do not state the daily cap from memory. Read GET /api/challenge/submission/quota and quote its limit / used / remaining; the limit is a platform setting and has changed before.
• Do not kill running agents to "free a slot" without checking the job list first; you may be killing the agent that's about to finish your last case.
• Do not retry a 4xx as if it were a network blip — see the diagnostic heuristic at the top of this file.
• Do not poll /result for status — it returns only {tasks, total}. A loop that branches on .status there never terminates. Use detailed_status from GET /api/challenge/jobs.

Reference

For the WS wire protocol (dial, control frames, binary frame layout, state machine, reconnect, heartbeat), see the appendix of challenge-run-agent — it is inlined there in full. Most contestants don't need it; ./scripts/tunnel.sh from the inference repo already implements it. For the observation/action payload schema inside the data frames, see challenge-inference-protocol.