Deep Analysis

Deep Analysis

Deep analysis of game logs — reads all raw log files, traces bugs to source code, and files detailed issues. For quick triage, use /fast-analysis instead.

Workflow

Step 1: Select the game

Determine which game to analyze:

• If the user specified a game ID (e.g. game_20260211_080409), use that.

• If the user said "most recent" or similar, find the latest:

  uv run python scripts/list_recent_games.py
  

• If the user mentioned a config name (e.g. "round-robin-commander", "jumpstart-dumb", "modern-staller"), use the corresponding symlink:

  uv run python scripts/list_recent_games.py --config {config}
  

  where {config} might be round-robin-commander, jumpstart-dumb, modern-staller, etc. Check what symlinks exist with --symlinks.

• If no game specified at all, find the most recent unanalyzed game:

  uv run python scripts/analysis/find_unanalyzed.py --type deep --count 1
  

Set GAME_DIR=~/.mage-bench/logs/{game_id}.

If the full log directory doesn't exist but the game export exists in website/public/games/ (.json or .json.gz), tell the user the full logs aren't available and offer to run a fast analysis from the export instead. Stop here unless the user wants the fast analysis.

Step 2: Bootstrap from gz (if available)

If website/public/games/${GAME_ID}.json or .json.gz exists (either on the current branch or generatable from the logs), extract a quick overview before diving into raw logs:

uv run python scripts/game_gz_bootstrap.py ${GAME_ID}

This gives you a roadmap — you'll know which players had errors, roughly when, and what to look for in the raw logs.

Bootstrap caveat: game_gz_bootstrap.py currently overcounts failed tool calls because it substring-matches normal fields like required, and its auto-export fallback still checks ~/mage-bench-logs instead of ~/.mage-bench/logs. Treat its failure count as advisory and run uv run python scripts/export_game.py ${GAME_ID} manually if the export is missing.

Check the errors array first: The export may contain an errors field with critical issues surfaced from the per-player error logs (loop detection, uncaught exceptions, short ID collisions). These are high-signal bug indicators — always check and call them out before diving into raw logs.

Check for existing annotations: If the export has an annotations array, blunder analysis has already been run. Reference these annotations to guide your investigation — they identify specific decisions that were likely mistakes and explain why.

Retry caveat: if an annotation claims a timeout/default choice on GAME_CHOOSE_CHOICE, verify it against raw *_llm.jsonl. scripts/export_game.py can currently record the first failed choose_action attempt and drop the later successful retry into a blank follow-up decision, which makes the annotation look like a timeout when the model actually recovered.

Crash-accounting caveat: if a player's export summary says toolCallsFailed=0 but the game clearly ended on a bad tool call, compare the tail of *_llm.jsonl and *_pilot.log. A final llm_response with no matching tool_call usually means the MCP request crashed before it could be logged. Those pilot-only crashes currently do not surface into export errors unless they also hit *_errors.log.

Check the decisions array: If the export has a decisions array, use extract_decisions.py to view structured decision records with board state, available choices, reasoning, and what happened next. Pass the export path, not just the game ID (for example: uv run python scripts/analysis/extract_decisions.py website/public/games/${GAME_ID}.json). This is often more useful than manually correlating events across log files.

Step 3: Read game metadata

Read config.json and game_meta.json — understand who played, what models/decks were used, and the game outcome (winner, turn count, life totals).

Step 4: Check existing issues

uv run python scripts/query_issues.py

Step 5: Analyze log files in parallel

Use parallel agents to analyze different log types simultaneously.

Pay particular attention to chat messages — they're the most human-readable signal about what went wrong. XMage sends game-state information, error messages, and rule explanations through chat. Players also chat when confused or stuck. Always read chat messages early and use them to guide your investigation of other log files.

• Chat messages: Extract player_chat events from game_events.jsonl (jq 'select(.type=="player_chat")' game_events.jsonl). Look for XMage system messages about illegal actions, failed spell resolutions, mana payment problems, and rule enforcement. Look for player messages that reveal confusion or frustration. Chat messages often point directly at the root cause before you even open error logs.
• Error logs: Read *_errors.log files. Look for Java exceptions (NPE, IndexOutOfBounds, ClassCast), MCP tool failures, and stack traces. Note the exact filename and line numbers.
• Pilot logs: Read *_pilot.log files. Look for LLM decision failures, repeated tool call patterns (loops), models sending wrong parameters, empty responses, and context trimming warnings. Pay close attention to what models complain about in their reasoning/thinking traces — when a model says "this doesn't make sense", "the tool returned wrong data", or "why can't I cast this", those are often smoking guns for real platform bugs rather than model confusion. Also check for personality infection: if the reasoning/thinking is full of in-character narrative (dramatic monologues, villain speeches, valley-girl speak) instead of game analysis, the chat personality is bleeding into gameplay decisions. Note the personality and flag affected decisions.
  • If a model appears to hallucinate IDs or misunderstand a prompt, compare the exact rendered tool text in *_llm_trace.jsonl against the structured MCP/export payload. This catches renderer bugs where the raw JSON has the needed field (for example incoming_attackers) but the prompt text shown to the model omits it.
  • If *_llm_trace.jsonl tells the model pass with answer=false, localize that stale instruction to the pilot state-bridge text in puppeteer/src/puppeteer/pilot.py, not to the live bridge tool schema.
  • If a triggered ability prompt only shows Stack: [Ability -> {'name': 'Player (you)', ...}], compare *_llm.jsonl / *_llm_trace.jsonl against exported snapshots. If the export has source_card / ability_text, localize the bug to bridge stack serialization plus decision_renderer.py, not the underlying game state.
  • If a pilot exits before any LLM activity with Bridge MCP HTTP server did not start ... (rc=1), inspect the embedded log tail or matching *_mcp.log before blaming the model. A common pre-game failure is the Java bridge dying during startup (for example BridgeClient.loadDeck() rejecting a generated .dck file such as line 1 NAME:...), so the HTTP port never opens.
• Bridge logs: Read *_bridge.jsonl files. Look for repeated identical MCP calls (loop signatures), failed actions, "Index out of range" errors, and action sequences that suggest confusion (e.g., cast → cancel → cast → cancel).
  • If GAME_CHOOSE_ABILITY shows descriptions like 1. ... but respond_with still says choice=0, choice=1, etc., trace the mismatch through AbilityPickerView plus BridgeCallbackHandler before classifying it as model misuse.
  • If a pilot calls pass_priority(until="stack_resolved") and then stalls, compare *_llm.jsonl, *_bridge.jsonl, and *_mcp.log. A final GAME_SELECT followed only by GAME_UPDATE / GAME_UPDATE_AND_INFORM plus long pendingAction=false waits means the bridge only re-checks stack_resolved while an actionable callback is pending.
  • For stale-choice race monitoring, use the Grep tool for choose_action out-of-range diagnostic in the game directory's *_mcp.log files and classify each hit:
    • likely model misuse: last_choices_response=boolean with index>=0, or negative index
    • suspicious race: last_choices_count >= 0 with tiny last_choices_age_ms and rapid action-type flips
  • Use the Grep tool (not bash rg) for searching log files:
    • choose_action out-of-range diagnostic in $GAME_DIR with glob *_mcp.log
    • Index .* out of range in $GAME_DIR with glob *_errors.log or *_pilot.log
  • For output schema validation errors (OpenAI structured outputs), grep for Invalid structured content returned by tool in *_pilot.log. These mean the bridge returned data that doesn't match the tool's output schema — the action succeeds server-side but the model gets an error. See doc/investigating-game-logs.md for details.
• Game events: Read game_events.jsonl. Look for stalls (long gaps between events), excessive auto-passes, turn timeouts, and game flow anomalies. For targeted investigation, use scripts/analysis/toolbox/game_timeline.py from the export — it supports --turns, --player, --mana, and -v flags to drill into specific turns or mana behavior. Caveat: some v7 exports omit snapshots[].ts, which currently makes game_timeline.py --turns ... misclassify every event as the final turn or print zero events. Fresh v8 exports without blunder annotations can also crash game_timeline.py with missing annotations because it currently uses the post-annotation loader. Sanity-check the output before relying on that filter; if it crashes, fall back to game_events.jsonl or extract_decisions.py.

Step 6: Cross-reference findings

A single bug often shows up across multiple log files. For example, an NPE in error logs corresponds to a failed tool call in bridge logs and a confused retry loop in pilot logs. Group these into one issue, not three.

Step 7: Distinguish code bugs from model issues

• Code bugs (file issues — the specimen appears to be broken): NPEs, wrong tool behavior, missing error handling, incorrect game state reporting — these need code fixes in Java or Python.
• Model behavior (note but don't file unless extreme): Passive play, bad threat assessment, suboptimal targeting — these are model quality issues. Only file if a model is completely non-functional (e.g., never plays spells, always passes).
• Personality infection (file as P3, label pilot): Check whether the player's chat personality is bleeding into their internal reasoning/thinking traces, causing worse gameplay decisions. This is a known antipattern where dramatic or expressive personalities (dramatist, villain, valley-girl, philosopher, etc.) cause the model to spend reasoning tokens on in-character narrative instead of board state analysis. Symptoms: reasoning full of dramatic monologue instead of game analysis; the model narrating intended plays in-character but timing out or picking wrong actions; the model interpreting normal game prompts as adversarial because the personality frames things dramatically (e.g. calling the game engine "broken" or "gaslighting"). If you see this, file an issue noting the personality, the specific decisions affected, and how the in-character reasoning led to the bad play. Analytical personalities (spike, analyst, detective) are generally clean.
• Toolset mismatches (file as P3): Look for tools that specific models are consistently hopeless with — always calling with wrong params, getting confused by responses, or wasting context on. Flag these as candidates for toolset changes in presets.json (each preset references a named toolset from toolsets.json). Note the model and the problematic tool so we can track patterns across games and decide whether to revoke that tool from weaker models' toolsets or design simpler alternative tools.
• Already handled (skip): Transient API errors with successful retries, empty responses caught by retry logic, one-off mistakes the model recovers from.

Step 8: Trace bugs to source code

For each code bug, read the relevant Java/Python files to identify the exact line and root cause. Include in the issue:

• The game log path: ~/.mage-bench/logs/game_YYYYMMDD_HHMMSS/
• Specific log files and approximate line numbers where the bug manifests
• The source code file and line where the fix should go (e.g., BridgeCallbackHandler.java:1407)
• A brief description of the root cause and suggested fix direction

Step 9: Create issue files

Before filing a new issue, check whether the bug has already been fixed since the game was played. Compare the game date against recent commits:

git log --oneline --since="YYYY-MM-DD" origin/master  # date of the game

If a commit clearly fixes the bug, skip filing the issue. If unsure, file it and note the possibly-relevant commit in the description.

Create issue files in issues/:

• Filename: issues/p{priority}-short-kebab-summary.json5
• Use issues/blocked-short-kebab-summary.json5 only when Gregor explicitly says the issue has manual preconditions; those files should also include "blocked": true

{
  "title": "Short summary",
  "description": "Full description with root cause analysis.\n\nEvidence:\n- ~/.mage-bench/logs/game_.../Player_errors.log: NPE at line 42\n- ~/.mage-bench/logs/game_.../Player_bridge.jsonl: repeated cast-cancel pattern\n\nSource: BridgeCallbackHandler.java:1407 — cv.getDisplayName() returns null\n\nSuggested fix: null-guard displayName before passing to StringBuilder",
  "status": "open",
  "priority": N,
  "type": "task",
  "labels": ["relevant-labels"],
  "created_at": "YYYY-MM-DDTHH:MM:SS.000000-08:00",
  "updated_at": "YYYY-MM-DDTHH:MM:SS.000000-08:00"
}

Priority guide:

• P1: Crashes or bugs that break core game actions (NPEs during targeting, spells fizzling due to code bugs)
• P2: Bugs causing major waste (infinite loops, stalling, repeated errors that block a player)
• P3: Suboptimal tool behavior or missing features (bad descriptions, missing info in prompts)
• P4: Minor issues (cosmetic, transient, or rare edge cases)

Labels: bridge, puppeteer, pilot, spectator

Step 10: Focus on a single game

Only analyze the game selected in step 1. Do not look at older games — each analysis run should be scoped to one game to keep context focused and output actionable.

Step 11: Present summary

Present a summary of all issues created, grouped by priority. For model-only issues, mention them in the summary but note they don't need code fixes.

Step 12: Create reusable analysis scripts

If you need to write any non-trivial log analysis logic (more than a simple jq one-liner), create a Python script in scripts/analysis/toolbox/ rather than writing throwaway one-off code. Run these scripts with uv run python scripts/analysis/toolbox/your_script.py. These scripts accumulate over time into a reusable analysis toolkit. Check what already exists in scripts/analysis/toolbox/ before creating something new — you may be able to reuse or extend an existing script. Key scripts already available: game_timeline.py, mcp_errors.py, mana_tapping.py, extract_decisions.py (note: extract_decisions.py is in scripts/analysis/, not toolbox/).

Step 13: Document investigation tricks

If you discovered any useful jq queries, grep patterns, cross-referencing techniques, or other tricks for investigating game logs during this analysis, append them to doc/investigating-game-logs.md. Don't duplicate what's already there — read the file first.

Step 14: Log the analysis

Create a file in doc/claudes/analyses/deep/ for the game analyzed (see doc/claudes/analyses/README.md for the template). This marks the game as deep-analyzed so future runs skip it.

Step 15: Update this skill

If you discovered new recurring patterns, useful analysis techniques, broken scripts, or better workflows during this run, update this file before finishing. This skill improves over time as more games are analyzed. Examples of things to add:

• New bug classes or error patterns to search for (add to Step 5 log analysis guidance)
• New model error patterns that are clearly not platform bugs
• Scripts that are broken or have known limitations
• Workflow improvements (e.g. better parallelization, useful cross-referencing techniques)
• New analysis scripts you created in scripts/analysis/toolbox/

Do not add issue-specific bug notes to the skill if you already filed them in issues/. The skill should capture reusable investigation technique, tooling caveats, and durable workflow guidance, not duplicate the issue tracker.