# Bit Exact Sim Validation

> Validate a port of a deterministic external system by building a per-step diff harness against captured authoritative state. The "trust anchor test" pattern — one regression test that asserts your port reproduces the original system's output exactly, plus a per-step diff harness for when that test fails. Use when porting a PRNG, game engine, financial model, protocol decoder, or any deterministic state-evolving system.

- Skill: `abedegno/bit-exact-sim-validation` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add abedegno/bit-exact-sim-validation`
- Raw SKILL.md: https://api.skillmd.com/api/skills/abedegno/bit-exact-sim-validation/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: abedegno (https://skillmd.com/u/abedegno)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/abedegno/bit-exact-sim-validation

---


# Bit-Exact Simulator Validation

When you've ported a deterministic external system (a game engine, a PRNG, a settlement model, a protocol decoder) and want to know your port is correct, the gold standard is **bit-exactness against captured authoritative state**. This skill is the discipline: one trust-anchor test that asserts the port matches the original on a representative captured session, plus a per-step diff harness for when that test fails.

The work this saves is enormous: most porting bugs are silent — your port produces "plausible-looking" output that differs from the original in subtle ways and you discover months later when downstream code drifts. Bit-exact validation catches those at port time.

## When this skill applies

- The system you're porting is **deterministic** (same input + same state → same output, always).
- You can **capture authoritative state** from the original — wire traffic, log files, recorded sessions, anything that gives you ground truth.
- You're going to use the port downstream — there's no point if you'd accept "approximately right".

If the system is non-deterministic (involves network randomness, timing, user input) you need a different validation approach (statistical equivalence, fuzz testing) — not this skill.

## The pattern

### The trust anchor test

A single test that loads a captured session, replays it through your port, and asserts the port matches the original output exactly. Pseudocode:

```python
def test_trust_anchor():
    """If this fails, the port is wrong and everything downstream is unreliable."""
    captured = load_captured_session("fixtures/canonical-session.jsonl")
    result = replay_through_port(captured.inputs)
    assert result.final_output == captured.final_authoritative_output
    # Also assert per-step state where available
    for i, observed in enumerate(captured.per_step_states):
        assert result.per_step_states[i] == observed, (
            f"divergence at step {i}: port produced {result.per_step_states[i]!r}, "
            f"original produced {observed!r}"
        )
```

This test is your **first line of defence**. It's not a unit test; it's an integration test against captured ground truth. If it passes, downstream is trustworthy. If it fails, you have a bug; nothing downstream is reliable until you fix it.

### The per-step diff harness

When the trust anchor fails, you need to know *where* it diverged. A diff harness walks the captured session step-by-step and prints the first divergence with full context:

```python
def diff_against_capture(captured):
    """Walk every step; print the first state mismatch with surrounding context."""
    state = port_initial_state(captured.seed)
    for i, (action, observed_after) in enumerate(zip(captured.actions, captured.states)):
        port_after = port_apply(state, action)
        if not deep_equal(port_after, observed_after):
            print(f"DIVERGENCE at step {i}")
            print(f"  action: {action!r}")
            print(f"  pre-state:")
            print_state(state)
            print(f"  port produced:")
            print_state(port_after)
            print(f"  original produced:")
            print_state(observed_after)
            print(f"  diff:")
            print_diff(port_after, observed_after)
            return i
        state = observed_after  # advance using authoritative state
    return None  # passed
```

Key design: **advance using the authoritative state, not the port's output.** That way one bug at step N doesn't cascade into apparent bugs at steps N+1, N+2... You'll see exactly one divergence; the rest of the trajectory is faithfully measured.

## Workflow

### Phase 1: Capture authoritative state

Build a way to record ground truth from the original system. Sources:

- **Wire traffic** — if the system is a service you can talk to. Use [`android-mitm-setup`](../android-mitm-setup/SKILL.md) for mobile apps, or browser DevTools for web. Save full sessions with `mitmdump -w session.flows`.
- **Log files** — if you can configure the original system to log per-step state.
- **Driven sessions** — a WS client or scripted browser that drives the system through a known sequence and captures everything (see [`closed-loop-live-demo`](../closed-loop-live-demo/SKILL.md) for the WS pattern).

What to capture per session:

- **Initial inputs**: seed, config, starting state.
- **Sequence of actions**: every input that mutated state.
- **Per-step state**: after each action, the full state. (Or whatever subset you can observe — final state + intermediate cumulative outputs is enough to start.)
- **Final output**: the score, the result, the cumulative score, whatever the trust-anchor compares.

Save as one file per session, in a stable format (JSONL with one event per line works well).

### Phase 2: Write the trust anchor test

The test goes in your test suite. It MUST be green before any other test you write is meaningful.

```python
# tests/test_trust_anchor.py
def test_validate_against_canonical_capture():
    """Load fixtures/canonical-session.jsonl, replay through port, assert
    output exactly matches the captured authoritative output. If this is
    red, fix it before believing any other test in this module."""
    fixture = REPO_ROOT / "fixtures" / "canonical-session.jsonl"
    report = validate_against_capture(fixture)
    assert report.passed, report.first_divergence_description
    assert report.final_score_port == report.final_score_authoritative
```

When the test is green, you have an evidence-backed claim that the port reproduces the original on at least one representative session. Capture more sessions to broaden coverage.

### Phase 3: Build the diff harness

The diff harness is a standalone script — `scripts/play_and_diff.py` is a good name — that:

1. Connects to the original (or loads a fixture).
2. Drives a session, capturing per-step state.
3. Replays the same inputs through the port.
4. Compares step-by-step, prints the first divergence with rich context.

The script's output is the input to debugging. Make it verbose: full pre-state, full post-state-of-port, full post-state-of-original, structured diff.

```python
# scripts/play_and_diff.py — invocation shape
async def main():
    # Connect, drive, capture
    captured = await play_and_capture(target_config)
    save_json(captured, f"scratch/play-and-diff-{ts}.json")

    # Diff against port
    first_divergence = diff_against_port(captured)
    if first_divergence is None:
        print("✓ NO DIVERGENCES — port agrees with original every step")
    else:
        print(f"✗ Divergence at step {first_divergence}")
```

### Phase 4: Iterate to convergence

When the diff finds a divergence:

1. Look at the **first** divergence only. Don't try to interpret later steps — they're meaningless until step N is fixed.
2. The **shape** of the divergence often points at the bug class. See [`references/divergence-signatures.md`](references/divergence-signatures.md).
3. Fix the bug in your port. Re-run the diff. Either you find the next divergence (closer to the end) or you reach `0 divergences`.
4. When you reach 0 divergences on one session, capture another and repeat. Different sessions exercise different code paths.

When several independent sessions all show 0 divergences across hundreds of state-mutations, the port is trustworthy.

## The trust anchor: lessons

### Empirical observation through a noisy channel isn't ground truth

A common failure mode: you "validated" your port by watching it run alongside the original and noting they "look the same" by eye. They don't look the same — your eye is doing massive data reduction. **Real validation requires structured per-cell comparison.**

A canonical example from one project: a rule about when a special tile spawns was learned by observing pixel animations in screen captures. The "rule" had ~41% observed accuracy — but the true rule had no such constraint, the 41% was classifier noise. The wrong rule survived for months because the validation channel (eyeballing animation) couldn't distinguish it from the right rule. Once the WS-script diff (skill E + this one) was built, the actual rule fell out in one session.

The lesson: **build the bit-exact channel.** Eyeballing is not validation.

### The right tool sometimes already exists in the repo from earlier work

The same project's WS client was originally built for security recon — to send malformed requests at the server and observe responses. Months later, when sim-vs-server bit-exactness became the bottleneck, that WS client was the exact piece needed. The lookup that worked was *"do I have a way to read authoritative per-step state?"* — not *"do I have a sim-divergence debug tool?"*

When stuck on validation, look across your repo for any code that talks to the live system. Often there's reusable infrastructure.

### Optimisation can make the underlying bug visible

If your port has a rare bug that triggers only on edge-case states, you may not hit it until you run **many** sessions. Optimising the port to run fast enough that you can drive 10 sessions per hour will surface the bug that "never happens".

The flip side: the bug was there the whole time; optimising just made it observable. "Make it fast" and "find the rare bug" are sometimes the same project.

### Validate before fixing

When something looks broken — "my port doesn't match production" — your first instinct will be to debug the port. Resist it. **First, run the diff harness and confirm the port IS wrong.** Sometimes the symptom you noticed has a different cause (a CV race, a stale capture, a timing issue). 15 minutes of diff-running can save hours of chasing the wrong bug.

## Divergence signatures

The shape of the divergence often tells you the bug class. See [`references/divergence-signatures.md`](references/divergence-signatures.md) for examples. As a teaser:

- **Many scattered wrong cells** → bug in the state-mutation rule itself (mark detection, scoring, etc.).
- **Top-of-one-column wrong, refilled values only** → bug in the PRNG refill order, or a CV / timing race.
- **One bit consistently wrong across all cells** → endianness or sign-extension bug.
- **First step matches, second step wrong** → PRNG state advanced by wrong amount.
- **Final score wrong by an integer multiple** → scoring multiplier off by one.

## Common pitfalls

- **The trust anchor test passes, but downstream code still misbehaves.** The captured session doesn't exercise the edge case your downstream code hits. Capture more sessions, including ones that exercise unusual paths.
- **The diff harness shows different divergences each run.** Either the original isn't actually deterministic (recheck), or your port has uninitialised state (check for `default = None` slots).
- **You "fix" a divergence and a new one appears 5 steps later.** Two interacting bugs — common when you've been guessing fixes rather than reading the divergence shape. Slow down; read the diff.
- **Captures contain sensitive data.** Sessions include tokens, account IDs, sometimes PII. Don't commit raw captures to a public repo; commit redacted minimal fixtures.

## References

- [`references/divergence-signatures.md`](references/divergence-signatures.md) — patterns in the shape of wrong state and what they typically mean.

## Pairs with

- [`android-mitm-setup`](../android-mitm-setup/SKILL.md) — capture authoritative state from mobile traffic.
- [`closed-loop-live-demo`](../closed-loop-live-demo/SKILL.md) — same WS infrastructure that drives the live demo also captures sessions for this skill.
- [`time-window-seed-bruteforce`](../time-window-seed-bruteforce/SKILL.md) — if the system is seeded and the port doesn't match initial state, your seed recovery may be wrong.
- [`negative-result-branches`](../negative-result-branches/SKILL.md) — when you can't fix a divergence, document the attempt on a branch.