# Closed Loop Live Demo > A single command that runs an offline prediction end-to-end against a live service and reports whether the prediction matched reality. The proof-of-claim pattern — offline benchmarks always look good, live closed-loop tests catch the gap between sim and reality. Use when you have a deterministic port (validated via bit-exact-sim-validation) and need to prove it predicts the production system's behaviour. - Skill: `abedegno/closed-loop-live-demo` (Agent Skill) - Install (CLI): `npx skillmds@latest add abedegno/closed-loop-live-demo` - Raw SKILL.md: https://api.skillmd.com/api/skills/abedegno/closed-loop-live-demo/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: abedegno (https://skillmd.com/u/abedegno) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/abedegno/closed-loop-live-demo --- # Closed-Loop Live Demo After you've ported a deterministic system and validated it bit-exactly against captured sessions ([`bit-exact-sim-validation`](../bit-exact-sim-validation/SKILL.md)), the next-level claim is: "given a fresh session, my port can predict the live service's outcome before the session even runs." This skill is the **single command** that proves it. The framing matters. Offline benchmarks always look good. Captured-session replays look good. Predicting a live session you've never seen, in advance, end-to-end, in one command — that's the demonstration that forces honesty. ## When this skill applies - You have a port of a deterministic external system. - The system exposes a way to start a session that you can drive programmatically (HTTPS REST + WebSocket is the common shape). - The system ships enough state to the client at session-start that your port can predict the outcome (seed + config + ruleset). - You can observe the actual outcome from the wire (cumulative score in clear msgpack, final-state message, etc.). If the system requires real-time user interaction that's hard to script, the closed-loop demo is harder to build but the principle is the same: drive it deterministically, capture the outcome, compare. ## The shape One Python entry point. Inside it, the canonical sequence: ``` 1. matchmake_post(entry_token) -> reservation 2. ws_connect(reservation.url) 3. handshake(ws) 4. send "request prepare" frame 5. recv "prepare" frame -> extract seed, config, ruleset 6. predicted = run_local_port(seed, config, planned_actions) 7. submit planned_actions (one-by-one or as a batch) 8. tail every state-mutation event; record cumulative score from each 9. recv "game over" frame or wait for the configured duration 10. actual = last_observed_cumulative_score 11. assert actual == predicted (or print delta) 12. write proof artifact: {seed, predicted, actual, n_events, match: "EXACT" | "Δ=..."} ``` The artifact is a small JSON file with a timestamp. Commit a few of them to your repo as evidence — they're tiny and they're the receipts. ## Example skeleton ```python import asyncio, json, time from pathlib import Path from curl_cffi import requests as cffi from websockets.asyncio.client import connect as ws_connect async def closed_loop_demo(launch_url): """Single command. Talks to the live service, predicts, drives, compares.""" # 1. Matchmake HTTPS POST to reserve a session entry_token = extract_token_from_launch_url(launch_url) reservation = matchmake(entry_token) # POST → JSON # 2-3. WebSocket connect + protocol handshake async with ws_connect(reservation["ws_url"], compression=None) as ws: await handshake(ws, reservation["session_id"]) # 4-5. Request initial state, parse seed await send_frame(ws, "REQUEST_PREPARE") prepare = await recv_frame(ws, type="PREPARE") seed = prepare["seed"] config = prepare["config"] # 6. Run local port actions, predicted_outcome = predict(seed, config, max_actions=30) # 7. Submit actions (one at a time, or as a batch — both work) for action in actions: await send_frame(ws, "ACTION", action) # 8. Tail every state-mutation event; record cumulative score events = [] deadline = time.monotonic() + 120.0 # 2-min budget while time.monotonic() < deadline: try: ev = await asyncio.wait_for(recv_any_frame(ws), timeout=2.0) except asyncio.TimeoutError: break if ev["type"] == "STATE_UPDATE": events.append(ev) if ev["type"] == "GAME_OVER": break actual_outcome = events[-1]["cumulative_score"] if events else None # 9-11. Compare match = "EXACT" if actual_outcome == predicted_outcome else f"Δ={actual_outcome - predicted_outcome}" print(f" seed: {seed}") print(f" predicted: {predicted_outcome:,}") print(f" actual: {actual_outcome:,}") print(f" events observed: {len(events)}") print(f" match: {match}") # 12. Write proof artifact artifact = { "seed": seed, "predicted_score": predicted_outcome, "actual_score": actual_outcome, "event_count": len(events), "match": match, "timestamp": time.time(), } out = Path(f"recon/closed-loop-proof-{int(time.time())}.json") out.parent.mkdir(parents=True, exist_ok=True) out.write_text(json.dumps(artifact, indent=2)) print(f" proof artifact written to: {out}") if __name__ == "__main__": import sys asyncio.run(closed_loop_demo(sys.argv[1])) ``` Invocation: ```bash python -m recon.closed_loop_demo "https://launch.example.com/play/" ``` ## What "EXACT" looks like ``` $ python -m recon.closed_loop_demo "https://launch.example.com/play/abc123" seed: 1780433928329 predicted: 58300 actual: 58300 events observed: 45 match: EXACT proof artifact written to: recon/closed-loop-proof-1780434060.json ``` That's the receipt: a fresh session, a never-before-seen seed, an outcome predicted from the seed alone, an actual outcome from the live service, matching exactly across 45 state-mutation events. ## What "near-exact" looks like ``` seed: 1780434360614 predicted: 107800 actual: 107500 events observed: 81 match: Δ=-300 ``` A delta means your port and the live system disagree about one specific event. Often it's a single rule that misfires in a rare configuration. Feed the session into your diff harness ([`bit-exact-sim-validation`](../bit-exact-sim-validation/SKILL.md)) to localise. ## Practical notes ### WebSocket library choice `websockets` (the Python library) is the standard. Two settings worth knowing: - `compression=None` — many production WS servers don't negotiate `permessage-deflate`; if you don't disable client-side, you get protocol errors mid-session. - `max_size=None` or `max_size=16*1024*1024` — server frames can exceed the default 1MB limit on busy sessions. ### Origin and Referer The matchmake endpoint and the WS upgrade often check `Origin` and `Referer`. Set these to whatever the production page uses (you can read it from your mitm capture or DevTools' Network panel). Mismatch → 502 from the upgrade. ### TLS-fingerprint WAF on the matchmake POST The matchmake endpoint is often WAF'd. Use `curl_cffi` with `impersonate="chrome"` for that HTTP POST — see [`mobile-auth-replay`](../mobile-auth-replay/SKILL.md) for the full pattern. The WS upgrade itself is typically NOT WAF'd because WAFs that inspect handshakes can't reliably parse WS upgrade negotiation. ### Cumulative score on the wire In many protocols (Colyseus, custom binary, msgpack-over-WS), the **cumulative score lives in plaintext on per-step state-mutation events**, even when the room state itself is schema-encoded. Reading the score doesn't require implementing the schema decoder — just decode the relevant msgpack field on each state event. ### Entry-token validation may be lax Surprisingly often, the matchmake endpoint accepts entry tokens it shouldn't: - 64-zero placeholder strings work. - Same token can be replayed many times. - Tokens issued for one user/session work for arbitrary other users. This is **distinct from the launch-page endpoint** which is usually stricter. If you find your matchmake call works with a known-invalid token, that's worth a private disclosure to the operator. ### Re-usable launch URLs In some implementations, a launch URL keeps reserving fresh sessions long after the original "consume" event. You can re-fire the same URL repeatedly for testing — saves credentials and avoids polluting your account state. Confirm against the specific system before relying on this; it varies by operator. ## Why this matters more than offline benchmarks Offline: - Captured session + my port = match. ✓ (But the capture is what it is — I didn't predict it, I replayed it.) - Greedy beats Random on synthetic seeds. ✓ (Both are running in my own simulator. Of course they agree.) Live closed-loop: - Fresh seed I've never seen, predicted offline, then the live service confirms in real time. ✗ if my port is wrong, ✓ if it's right. A green offline test only proves your port is consistent. A green live closed-loop test proves it's correct against reality. This is the demo to lead with when explaining the project to someone. Tables of offline numbers don't move people. A single live command that says EXACT does. ## Disclosure implications A closed-loop demo against a live service can demonstrate exploitability of an information-disclosure issue (e.g. "the system leaks the seed, and given the seed I can predict the outcome exactly"). When demonstrating to the operator: - Send the proof artifact JSON, not the running command. - Redact your specific session token / launch URL. - Note that the demo runs as your own user against your own session; it doesn't impersonate anyone or read other users' state. - Offer to retract the public demo if the operator wants disclosure-window time to deploy a fix. This skill produces evidence; how you use the evidence is the disclosure question, not a technique question. ## Pairs with - [`bit-exact-sim-validation`](../bit-exact-sim-validation/SKILL.md) — the port that produces `predicted` must be validated first. - [`mobile-auth-replay`](../mobile-auth-replay/SKILL.md) — if the matchmake endpoint requires an auth bearer, the auth-replay helper gets you the token. - [`android-mitm-setup`](../android-mitm-setup/SKILL.md) — observe the actual matchmake / WS protocol before scripting it. - [`time-window-seed-bruteforce`](../time-window-seed-bruteforce/SKILL.md) — if the seed isn't shipped on the wire, you may need to brute-force it from observed state.