ClaudePlayer
An AI agent that plays Game Boy games autonomously using Claude's vision capabilities and the PyBoy emulator.
Overview
ClaudePlayer connects Claude to a Game Boy emulator. Each turn, Claude receives a screenshot of the current game frame, reasons about what to do, and sends button inputs back to the emulator. It maintains a structured memory system to track game progress (items, NPCs, locations, quests, stats).
Prerequisites
- Python 3.10+
- Pipenv
- An Anthropic API key
- A Game Boy ROM file (
.gb)
Setup
# Install dependencies
pipenv install
# Create .env with your API key
echo "ANTHROPIC_API_KEY=your_key_here" > .env
# Place a Game Boy ROM in the project directory
Configuration
Edit config.json:
| Key |
Description |
Default |
ROM_PATH |
Path to Game Boy ROM |
Required |
STATE_PATH |
Saved emulator state file |
null |
EMULATION_MODE |
"turn_based" or "continuous" |
"turn_based" |
MODEL_DEFAULTS.MODEL |
Claude model to use |
"claude-sonnet-4-5-20250929" |
MODEL_DEFAULTS.THINKING |
Enable extended thinking |
true |
RATE_LIMITS.RPM_THRESHOLD |
Requests per minute limit |
Configured in file |
RATE_LIMITS.TPM_THRESHOLD |
Tokens per minute limit |
Configured in file |
REDIS_LOGS |
Optional Redis logging config |
null |
Running
pipenv shell
# Default config
python play.py
# Custom config
python play.py --config my_config.json
# Create a saved emulator state
python emu_setup.py
Project Structure
| Path |
Purpose |
play.py |
Launcher script |
claude_player/main.py |
CLI entry point with arg parsing |
claude_player/agent/game_agent.py |
Main orchestrator: emulator init, game loop, coordination |
claude_player/interface/claude_interface.py |
Claude API communication and rate limiting |
claude_player/state/game_state.py |
Game state tracking (memory, goals, history) |
claude_player/tools/tool_setup.py |
Tool definitions (send_inputs, memory ops, etc.) |
claude_player/tools/tool_registry.py |
Tool registry system |
claude_player/config/config_loader.py |
Config file parsing |
claude_player/config/config_class.py |
Configuration data class |
claude_player/utils/game_utils.py |
Button input parsing, screenshot capture |
claude_player/utils/memory_reader.py |
Game memory/stats reader |
claude_player/agent/summary_generator.py |
Periodic game progress summarization |
config.json |
Runtime configuration |
Available Tools (for the agent during gameplay)
| Tool |
Description |
send_inputs |
Send button sequences to the emulator |
set_game |
Identify the current game |
set_current_goal |
Update the gameplay objective |
add_to_memory |
Store items, NPCs, locations, quests, mechanics, stats |
remove_from_memory |
Remove a memory entry |
update_memory_item |
Update an existing memory entry |
toggle_thinking |
Toggle extended thinking mode on/off |
How It Works
- PyBoy emulator loads the ROM and renders a frame
- The frame is captured as a screenshot and sent to Claude
- Claude analyzes the screen, reasons about the game state, and calls tools
send_inputs presses buttons on the emulator (A, B, Up, Down, Left, Right, Start, Select)
- In turn-based mode, the emulator only advances when the agent acts
- The agent maintains structured memory to track long-term game progress
- Periodic summaries compress the conversation history to stay within context limits
1---2name: claude-player3description: An AI-powered Game Boy emulator agent that uses Claude's vision and reasoning to autonomously play Game Boy games.4---56# ClaudePlayer78An AI agent that plays Game Boy games autonomously using Claude's vision capabilities and the PyBoy emulator.910## Overview1112ClaudePlayer connects Claude to a Game Boy emulator. Each turn, Claude receives a screenshot of the current game frame, reasons about what to do, and sends button inputs back to the emulator. It maintains a structured memory system to track game progress (items, NPCs, locations, quests, stats).1314## Prerequisites1516- Python 3.10+17- Pipenv18- An Anthropic API key19- A Game Boy ROM file (`.gb`)2021## Setup2223```bash24# Install dependencies25pipenv install2627# Create .env with your API key28echo "ANTHROPIC_API_KEY=your_key_here" > .env2930# Place a Game Boy ROM in the project directory31```3233## Configuration3435Edit `config.json`:3637| Key | Description | Default |38|-----|-------------|---------|39| `ROM_PATH` | Path to Game Boy ROM | Required |40| `STATE_PATH` | Saved emulator state file | `null` |41| `EMULATION_MODE` | `"turn_based"` or `"continuous"` | `"turn_based"` |42| `MODEL_DEFAULTS.MODEL` | Claude model to use | `"claude-sonnet-4-5-20250929"` |43| `MODEL_DEFAULTS.THINKING` | Enable extended thinking | `true` |44| `RATE_LIMITS.RPM_THRESHOLD` | Requests per minute limit | Configured in file |45| `RATE_LIMITS.TPM_THRESHOLD` | Tokens per minute limit | Configured in file |46| `REDIS_LOGS` | Optional Redis logging config | `null` |4748## Running4950```bash51pipenv shell5253# Default config54python play.py5556# Custom config57python play.py --config my_config.json5859# Create a saved emulator state60python emu_setup.py61```6263## Project Structure6465| Path | Purpose |66|------|---------|67| `play.py` | Launcher script |68| `claude_player/main.py` | CLI entry point with arg parsing |69| `claude_player/agent/game_agent.py` | Main orchestrator: emulator init, game loop, coordination |70| `claude_player/interface/claude_interface.py` | Claude API communication and rate limiting |71| `claude_player/state/game_state.py` | Game state tracking (memory, goals, history) |72| `claude_player/tools/tool_setup.py` | Tool definitions (send_inputs, memory ops, etc.) |73| `claude_player/tools/tool_registry.py` | Tool registry system |74| `claude_player/config/config_loader.py` | Config file parsing |75| `claude_player/config/config_class.py` | Configuration data class |76| `claude_player/utils/game_utils.py` | Button input parsing, screenshot capture |77| `claude_player/utils/memory_reader.py` | Game memory/stats reader |78| `claude_player/agent/summary_generator.py` | Periodic game progress summarization |79| `config.json` | Runtime configuration |8081## Available Tools (for the agent during gameplay)8283| Tool | Description |84|------|-------------|85| `send_inputs` | Send button sequences to the emulator |86| `set_game` | Identify the current game |87| `set_current_goal` | Update the gameplay objective |88| `add_to_memory` | Store items, NPCs, locations, quests, mechanics, stats |89| `remove_from_memory` | Remove a memory entry |90| `update_memory_item` | Update an existing memory entry |91| `toggle_thinking` | Toggle extended thinking mode on/off |9293## How It Works94951. PyBoy emulator loads the ROM and renders a frame962. The frame is captured as a screenshot and sent to Claude973. Claude analyzes the screen, reasons about the game state, and calls tools984. `send_inputs` presses buttons on the emulator (A, B, Up, Down, Left, Right, Start, Select)995. In turn-based mode, the emulator only advances when the agent acts1006. The agent maintains structured memory to track long-term game progress1017. Periodic summaries compress the conversation history to stay within context limits