Replay Batches

Debugging in-process replay failures

How to find and fix the class of bugs where a replay passes on its own but fails when run after another replay in the same process (the "in-process batch").

The bug class

ZC normally runs one quest/game per process. The in-process batch harness runs many replays back-to-back in one process to flush out global/static state that isn't reset on game start. A leftover value from game A bleeds into game B and desyncs it.

Every fix so far has the same shape: a file-static or global that is implicitly "fresh" at process start but never re-initialized when a new game begins. The fix is almost always a one-line reset in init_game_vars() (src/zc/zelda.cpp) or HeroClass::init() (src/zc/hero.cpp), next to the existing resets there.

Examples already fixed (good reference for the pattern):

• next_uid sprite counter (src/sprite.cpp) → sprite::reset_uid_counter()
• hero_newstep / hero_newstep_diag collision step (src/zc/hero.cpp)
• z3step, Hero.hitdir, Hero.holditem, climb_cover_x/y, scroll-from state
• palette_user_tinted script tint (a render-only leak — see below)
• subscreen_open, screen_states (already cleared via clear_screen_states())

Tools

Engine flags (build: cmake --build build --config Release -t zplayer, exe at build/Release/zplayer):

• -replay-batch <batch file>: run every replay in the batch file in one process. Each line is <absolute-path> [frame-limit]. Mode comes from a bare -assert / -replay / -update flag. Always pass -headless for hunts.
• -state-hash-log <file>: append per-frame FNV-1a hashes of game-state regions (hero, rng, globalRAM, objectRAM, sprites, viewport, ffc, screen, scroll) to a file. Also dumps HEROFIELDS (full HeroClass::debug_state_string()) at frame 0.

Runner (tests/run_replay_tests.py):

• --batch --seed N --frame 1000: shuffle all replays into batches (seeded) and batch-run each batch. Writes failing_batch_NN.txt for each failing batch and prints a re-run command.
• --replay-batch <file>: run exactly the replays in one batch file (in order, in one process) and report per-replay pass/fail with the failing frame. This is the reliable pass/fail signal.
• tests/replays_vary_seed.sh N ...: run N seeds, stop at first failure.

Scratch repros live in .tmp/failure_groups/ (gitignored, absolute paths). The durable regression suite is tests/replay_batches/*.txt — committed batch files with paths relative to the repo root, run by tests/test_replays.py::test_batches (it rewrites them to absolute paths and asserts every replay in each batch passes).

Workflow

1. Find a failing batch. python tests/run_replay_tests.py --batch --seed 6014 --frame 1000. Copy the failing_batch_NN.txt files somewhere safe.

2. Find which replay actually fails. --replay-batch failing_batch_NN.txt. The runner prints ❌ … failure on frame X. Note: a replay that fails solo is NOT an in-process leak — verify solo first and skip it if it fails alone.

3. Bisect the poisoning predecessor. Build a 2-line batch file: <predecessor> 1000 then <target> 1000. Try each predecessor (and combinations — some need two). Reproduce the failure with the runner, not by grepping engine output.

4. Pin the diverging subsystem with -state-hash-log. Run the target solo and poisoned, each with its own log, then diff the target's lines column-by-column to find the first frame and the first differing column:

   ./build/Release/zplayer -assert -headless -replay-batch solo.txt  -state-hash-log /tmp/s.log
   ./build/Release/zplayer -assert -headless -replay-batch pois.txt  -state-hash-log /tmp/p.log
   paste <(grep 'TARGET.zplay frame=' /tmp/s.log) <(grep 'TARGET.zplay frame=' /tmp/p.log) \
     | awk '{split($0,a,"\t");split(a[1],s," ");split(a[2],p," ");
             for(i=2;i<=length(s);i++) if(s[i]!=p[i]) print s[2],"DIFF:",s[i],"vs",p[i]}' | head
   

5. Drill to the exact field. If the hero/heroint column diverges, diff the frame-0 HEROFIELDS lines token-by-token (watch out: a naive grep '[a-z_]+=' drops camelCase fields like diagonalMovement). If the differing value isn't in debug_state_string, temporarily add the suspect field to it (or add a temp hash column to dump_state_hashes in src/zc/ffscript.cpp) and rebuild. Iterate until you see the exact leaked field.

6. Confirm before fixing. Always prove the field leaks with the instrument first. Many plausible suspects (climb_cover, hitdir, z3step) turned out to be real leaks that were not the cause of a given test — fixing them didn't flip the test. Confirm the field differs at the first divergence frame, fix it, and re-verify with the runner.

7. Fix by resetting the global/static on game start in init_game_vars() or HeroClass::init() (mind the replay_version_check(N) guard pattern used by z3step — it preserves old-replay behavior while fixing new ones). Add a comment explaining the leak. Rebuild and confirm the repro passes via the runner.

8. Add a regression batch file — REQUIRED for every in-process leak fix. Create tests/replay_batches/<name>.txt with the minimal <predecessor> <frame> / <target> <frame> lines that reproduced the failure, using paths relative to the repo root (e.g. tests/replays/playground/ghost_armos.zplay 200) and a leading # comment describing the leak it guards. test_batches picks it up automatically; run it to confirm it passes now (and ideally that it fails before your fix). This is what prevents the leak from silently coming back.

9. Commit each distinct leak as its own fix(zc): … commit (engine fixes go at the tail of history); commit the regression batch file with it.

Reading a .roundtrip file (what diverged, in replay terms)

When an -assert replay fails, the engine writes <name>.zplay.roundtrip next to the replay output (e.g. .tmp/test_results/<id>/<run>/batch_<g>/<name>.zplay.roundtrip); the runner also embeds it in the failure output. It is the most direct view of what diverged, in the replay's own event vocabulary — often more legible than a state-hash diff.

It is a side-by-side annotation (written in replay_save, src/zc/replay.cpp):

• The left column is what this run actually did (the recomputed record_log).
• Where a step differs from the recording, the line is annotated … « <step>. The right side of « is what should have happened — the recorded/expected step (replay_log).
• Trailing lines with a blank left and « <step> are recorded steps the run never reached (it diverged or ended early). A run that desyncs and quits early looks like a few «-only lines at the bottom.

The first « line is the first divergence. Read it as: left = what your (poisoned) run produced, right = what the correct (solo) run produced.

Step syntax (from print_step): M k v = metadata header; C <frame> g <hash> = per-frame framebuffer (gfx) hash; C <frame> <text> = engine comment (init_game, cont_game, getitem <name>, sfx <name>, screen-state, …); D <frame> [k] <btn/key> = press, U = release; Q <frame> <quit> = the Quit value was set. The diverging step usually names the subsystem that leaked — a menu action, an item grant, a Quit, a warp.

Worked example (terror_27 → hero_of_dreams): left showed D 1024 k F6 → Q 1024 1 → C 1025 init_game (the poisoned run's F6 press quit immediately); the « right side showed D 1024 k F6 / U 1025 k F6 / D 1035 Start / … (the recording's F6 opened a menu the player navigated and continued from). Conclusion: the F6 menu took the wrong path when poisoned → F6-menu availability state leaked from the previous quest.

Render-only leaks (game state matches, replay still fails)

If all -state-hash-log columns match but the replay still fails, the leak is in rendered output, not game state. Debug replays record a per-frame framebuffer hash (the g comments, via replay_step_gfx / hash_bitmap); the runner's "failure on frame X" can be a gfx-hash mismatch. The state-hash instrument can't see this by default.

To pin it, temporarily add a gfx column to dump_state_hashes that calls a helper wrapping hash_bitmap(framebuf) (expose one from replay.cpp). The framebuffer hash is computed on a 24bpp conversion, so it reflects the palette — index-based game state can match while colors differ.

Worked example (the palette_user_tinted fix): terror_of_necromancy_demo6_34, poisoned by enigma, had identical game state but a constant framebuffer diff from frame 1. The map viewer (ViewMap) made it visually obvious. Chain: gfx differs → palette (RAMpal) differs → nonblack=14 vs 234 (game palette blacked, GUI colors survive) → narrowed within init_game() to the step between loadfullpal and loadlvlpal → loadpalset/loadlvlpal route to tempgreypal instead of RAMpal (and restoreTint blacks RAMpal) when isUserTinted() is true → palette_user_tinted leaked true from enigma's script tint. Fix: isUserTinted(false) in init_game_vars().

Gotchas

• Don't trust grep unexpected on headless -replay-batch. -unexpected.png files are only written in non-headless / snapshot mode. Use the -state-hash-log diff, or the runner's --replay-batch pass/fail, as the source of truth.
• Some replays need two predecessors to reproduce; one alone won't poison.
• -snapshot and some report features reset per-replay in batch — use single-replay mode when you need them.
• A divergence in heroint (the comprehensive hero hash) with no visible/gameplay effect can be a harmless leaked field; confirm it actually causes the failure before committing.