replay-batches

name	replay-batches
description	How to find and fix the class of bugs where global state is leaked when loading multiple quests. For example, a replay test may pass on its own but fail when run after another replay in the **same process** (the "in-process batch"). The "--replay-batch" flag for "tests/run_replay_tests.py" is relevant.

Debugging in-process replay failures

How to find and fix the class of bugs where a replay passes on its own but fails when run after another replay in the same process (the "in-process batch").

The bug class

ZC normally runs one quest/game per process. The in-process batch harness runs many replays back-to-back in one process to flush out global/static state that isn't reset on game start. A leftover value from game A bleeds into game B and desyncs it.

Every fix so far has the same shape: a file-static or global that is implicitly "fresh" at process start but never re-initialized when a new game begins. The fix is almost always a one-line reset in init_game_vars() (src/zc/zelda.cpp) or HeroClass::init() (src/zc/hero.cpp), next to the existing resets there.

Examples already fixed (good reference for the pattern):

• next_uid sprite counter (src/sprite.cpp) → sprite::reset_uid_counter()
• hero_newstep / hero_newstep_diag collision step (src/zc/hero.cpp)
• z3step, Hero.hitdir, Hero.holditem, climb_cover_x/y, scroll-from state
• palette_user_tinted script tint (a render-only leak — see below)
• subscreen_open, screen_states (already cleared via clear_screen_states())

Tools

Engine flags (build: cmake --build build --config Release -t zplayer, exe at build/Release/zplayer):

• -replay-batch <batch file>: run every replay in the batch file in one process. Each line is <absolute-path> [frame-limit]. Mode comes from a bare -assert / -replay / -update flag. Always pass -headless for hunts.
• -state-hash-log <file>: append per-frame FNV-1a hashes of game-state regions (hero, rng, globalRAM, objectRAM, sprites, viewport, ffc, screen, scroll) to a file. Also dumps HEROFIELDS (full HeroClass::debug_state_string()) at frame 0.

Runner (tests/run_replay_tests.py):

• --batch --seed N --frame 1000: shuffle all replays into batches (seeded) and batch-run each batch. Writes failing_batch_NN.txt for each failing batch and prints a re-run command.
• --replay-batch <file>: run exactly the replays in one batch file (in order, in one process) and report per-replay pass/fail with the failing frame. This is the reliable pass/fail signal.
• tests/replays_vary_seed.sh N ...: run N seeds, stop at first failure.

Scratch repros live in .tmp/failure_groups/ (gitignored, absolute paths). The durable regression suite is tests/replay_batches/*.txt — committed batch files with paths relative to the repo root, run by tests/test_replays.py::test_batches (it rewrites them to absolute paths and asserts every replay in each batch passes).

Workflow

1. Find a failing batch. python tests/run_replay_tests.py --batch --seed 6014 --frame 1000. Copy the failing_batch_NN.txt files somewhere safe.

2. Find which replay actually fails. --replay-batch failing_batch_NN.txt. The runner prints ❌ … failure on frame X. Note: a replay that fails solo is NOT an in-process leak — verify solo first and skip it if it fails alone.

3. Bisect the poisoning predecessor. Build a 2-line batch file: <predecessor> 1000 then <target> 1000. Try each predecessor (and combinations — some need two). Reproduce the failure with the runner, not by grepping engine output.

4. Pin the diverging subsystem with -state-hash-log. Run the target solo and poisoned, each with its own log, then diff the target's lines column-by-column to find the first frame and the first differing column:

   ./build/Release/zplayer -assert -headless -replay-batch solo.txt  -state-hash-log /tmp/s.log
   ./build/Release/zplayer -assert -headless -replay-batch pois.txt  -state-hash-log /tmp/p.log
   paste <(grep 'TARGET.zplay frame=' /tmp/s.log) <(grep 'TARGET.zplay frame=' /tmp/p.log) \
     | awk '{split($0,a,"\t");split(a[1],s," ");split(a[2],p," ");
             for(i=2;i<=length(s);i++) if(s[i]!=p[i]) print s[2],"DIFF:",s[i],"vs",p[i]}' | head
   

5. Drill to the exact field. If the hero/heroint column diverges, diff the frame-0 HEROFIELDS lines token-by-token (watch out: a naive grep '[a-z_]+=' drops camelCase fields like diagonalMovement). If the differing value isn't in debug_state_string, temporarily add the suspect field to it (or add a temp hash column to dump_state_hashes in src/zc/ffscript.cpp) and rebuild. Iterate until you see the exact leaked field.

6. Confirm before fixing. Always prove the field leaks with the instrument first. Many plausible suspects (climb_cover, hitdir, z3step) turned out to be real leaks that were not the cause of a given test — fixing them didn't flip the test. Confirm the field differs at the first divergence frame, fix it, and re-verify with the runner.

7. Fix by resetting the global/static on game start in init_game_vars() or HeroClass::init() (mind the guard pattern used by — it preserves old-replay behavior while fixing new ones). Add a comment explaining the leak. Rebuild and confirm the repro passes via the runner.

Reading a .roundtrip file (what diverged, in replay terms)

When an -assert replay fails, the engine writes <name>.zplay.roundtrip next to the replay output (e.g. .tmp/test_results/<id>/<run>/batch_<g>/<name>.zplay.roundtrip); the runner also embeds it in the failure output. It is the most direct view of what diverged, in the replay's own event vocabulary — often more legible than a state-hash diff.

It is a side-by-side annotation (written in replay_save, src/zc/replay.cpp):

• The left column is what this run actually did (the recomputed record_log).
• Where a step differs from the recording, the line is annotated … « <step>. The right side of « is what should have happened — the recorded/expected step (replay_log).
• Trailing lines with a blank left and « <step> are recorded steps the run never reached (it diverged or ended early). A run that desyncs and quits early looks like a few «-only lines at the bottom.

The first « line is the first divergence. Read it as: left = what your (poisoned) run produced, right = what the correct (solo) run produced.

Step syntax (from print_step): M k v = metadata header; C <frame> g <hash> = per-frame framebuffer (gfx) hash; C <frame> <text> = engine comment (init_game, cont_game, getitem <name>, sfx <name>, screen-state, …); D <frame> [k] <btn/key> = press, U = release; Q <frame> <quit> = the Quit value was set. The diverging step usually names the subsystem that leaked — a menu action, an item grant, a Quit, a warp.

Worked example (terror_27 → hero_of_dreams): left showed D 1024 k F6 → Q 1024 1 → C 1025 init_game (the poisoned run's F6 press quit immediately); the « right side showed D 1024 k F6 / U 1025 k F6 / D 1035 Start / … (the recording's F6 opened a menu the player navigated and continued from). Conclusion: the F6 menu took the wrong path when poisoned → F6-menu availability state leaked from the previous quest.

Render-only leaks (game state matches, replay still fails)

If all -state-hash-log columns match but the replay still fails, the leak is in rendered output, not game state. Debug replays record a per-frame framebuffer hash (the g comments, via replay_step_gfx / hash_bitmap); the runner's "failure on frame X" can be a gfx-hash mismatch. The state-hash instrument can't see this by default.

To pin it, temporarily add a gfx column to dump_state_hashes that calls a helper wrapping hash_bitmap(framebuf) (expose one from replay.cpp). The framebuffer hash is computed on a 24bpp conversion, so it reflects the palette — index-based game state can match while colors differ.

Worked example (the palette_user_tinted fix): terror_of_necromancy_demo6_34, poisoned by enigma, had identical game state but a constant framebuffer diff from frame 1. The map viewer (ViewMap) made it visually obvious. Chain: gfx differs → palette (RAMpal) differs → nonblack=14 vs 234 (game palette blacked, GUI colors survive) → narrowed within init_game() to the step between loadfullpal and loadlvlpal → loadpalset/loadlvlpal route to tempgreypal instead of RAMpal (and restoreTint blacks RAMpal) when isUserTinted() is true → palette_user_tinted leaked true from enigma's script tint. Fix: isUserTinted(false) in init_game_vars().

Gotchas

• Don't trust grep unexpected on headless -replay-batch. -unexpected.png files are only written in non-headless / snapshot mode. Use the -state-hash-log diff, or the runner's --replay-batch pass/fail, as the source of truth.
• Some replays need two predecessors to reproduce; one alone won't poison.
• -snapshot and some report features reset per-replay in batch — use single-replay mode when you need them.
• A divergence in heroint (the comprehensive hero hash) with no visible/gameplay effect can be a harmless leaked field; confirm it actually causes the failure before committing.