| name | debug-desync |
| description | Root-cause a SnakeTron state-sync bug (phantom death, drift, ghost game, desync report) from captured traces — replay them deterministically, find the first divergent tick, and turn the bug into a local failing test. Use whenever a sync issue is reported from prod or observed in dev. |
Debugging a state-sync issue from traces
Read DEBUGGING.md at the repo root first — it explains the four sync
primitives (fingerprints, TickHash probes, stream_seq, flight recorders) and
the full runbook. This skill is the condensed procedure.
1. Collect evidence
- Server trace:
traces/game_<id>_server_*.jsonl (or $SNAKETRON_TRACE_DIR).
- Client trace:
traces/game_<id>_client_<user>_*.jsonl — auto-uploaded on
detected desyncs; otherwise ask the reporter to run
window.snaketronDebug.downloadTrace() in the browser console.
- If you only have one side, proceed anyway — a server trace alone can prove
or rule out engine nondeterminism; a client trace alone shows what the
client actually received.
2. Analyze
cargo run --bin trace_rca -- <server_trace> [<client_trace>]
cargo run --bin trace_rca -- <server_trace> <client_trace> --json
Trust the verdict as a starting hypothesis, then verify against the report
details:
TRANSPORT_LOSS: check missing_stream_seqs — which messages, at what
tick. Then ask: did the client detect the gap (Note records) and resync?
If not, the detection/resync path is the bug, not the loss itself.ENGINE_NONDETERMINISM: same inputs produced different states. Use the
first divergent tick to bisect the engine logic (common/src/game_state.rs tick_forward, event application order, native-vs-WASM differences). This
is the most serious verdict.CLOCK_DRIFT: client tick computation off; check Clock records and
command reschedule deltas.- Phantom-death report: look at command latency — a command whose
command_id_server.tick > command_id_client.tick was rescheduled because
it arrived past the 500 ms committed-lag window; the client showed the turn
at the requested tick, the server executed it later.
3. Freeze the bug as a test
cargo run --bin trace_rca -- <server_trace> --emit-test server/tests/repro_<desc>.rs
cargo test -p server --test repro_<desc>
Commit the trace fixture and the test with the fix.
4. Fix and generalize
After fixing, extend the chaos suite
(server/tests/sync_equivalence_test.rs) with the failure shape you found —
that suite simulates lossy/latent/jittery transport in-process and is the
regression barrier for this bug class. cargo test -p server must be green.
Live-system spot checks
- Client sync health in the browser console:
JSON.parse(engine.getSyncStatusJson()) — gap counts, probe mismatches,
needs_resync.
- Server logs: grep for
lagged, gap, resync, channel full — every
loss path now logs loudly.
