# Automation Mode

> Automation mode in FlatRedBall2. Use when an external agent (AI or script) needs to drive a running game: stepping frames, injecting input, querying entity state, or forcing entity values over stdin/stdout. Covers EnableAutomationMode, the NDJSON command protocol, reflection-based entity introspection, and optional RegisterStateProvider for derived state.

- Skill: `vchelaru/automation-mode-2` (Agent Skill)
- Install (CLI): `npx skillmds@latest add vchelaru/automation-mode-2`
- Raw SKILL.md: https://api.skillmd.com/api/skills/vchelaru/automation-mode-2/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: vchelaru (https://skillmd.com/u/vchelaru)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/vchelaru/automation-mode-2

---


# Automation Mode in FlatRedBall2

Automation mode lets an external process control a running game via NDJSON (newline-delimited JSON) over stdin/stdout — one command per line in, one response per line out. The primary use case is AI agents that need to observe and interact with the game the way Playwright interacts with a browser.

**Debug builds only.** `EnableAutomationMode()` is a no-op in Release. This is intentional — automation mode exposes internal state and allows arbitrary value forcing.

## Setup

```csharp
// Game.Initialize, after base.Initialize():
FlatRedBallService.Default.EnableAutomationMode();
```

The call itself does nothing unless `--frb-auto` is present in the command-line args. Ship the call unconditionally; the flag controls activation.

```
dotnet run -- --frb-auto
```

That's the entire required wiring. Entities and their public properties are auto-discovered — no per-property registration.

### Deterministic randomness

When automation activates, `FlatRedBallService.Random` is replaced with a deterministic `GameRandom` so recorded NDJSON sessions reproduce exactly. Pass an explicit seed to choose which run.

```csharp
FlatRedBallService.Default.EnableAutomationMode(seed: 1234);
```

| `--frb-auto` flag | `seed:` param | Result |
|---|---|---|
| absent | (any) | time-based — automation off, seed param ignored |
| present | omitted | seed `0` |
| present | `1234` | seed `1234` |

The seed is only applied when `--frb-auto` activates automation; without the flag, ship-time seed values have no effect on a normal run.

Game code that constructs its own `Random`/`GameRandom` is **not** seeded by the engine — route gameplay randomness through `FlatRedBallService.Random` (or derive your own seed from it) if you want it included in the deterministic run.

## Command Protocol

Each command is a JSON object terminated by `\n`. Each response is a JSON object on its own line, always containing `ok` and `frame`.

| Command | JSON |
|---------|------|
| Step N frames | `{"cmd":"step"}` or `{"cmd":"step","count":5}` |
| Key down/up | `{"cmd":"input","type":"key","key":"Space","down":true}` |
| Gamepad button | `{"cmd":"input","type":"gamepad","player":0,"button":"A","down":true}` |
| Gamepad axis | `{"cmd":"input","type":"axis","player":0,"axis":"LeftStickX","value":0.8}` |
| Cursor (screen px) | `{"cmd":"input","type":"cursor","x":120,"y":80,"primary":true}` |
| Cursor (world coords) | `{"cmd":"input","type":"cursor","x":0,"y":0,"space":"world","primary":true}` |
| Query active screen | `{"cmd":"query","target":"screen"}` |
| Query all entities | `{"cmd":"query","target":"entities"}` |
| Query one entity type | `{"cmd":"query","target":"Player"}` |
| Force a value | `{"cmd":"set","entity":"Player","prop":"X","value":100.0}` |
| Screenshot next Draw | `{"cmd":"record_next_screenshot","path":"shot.png"}` |
| Quit | `{"cmd":"quit"}` |

### Frame stepping

The game loop gates each frame on pending step commands. Without a `step` the game does not advance — it returns from `Update` early and suppresses Draw each tick.

Commands process **in the order they were sent**. A query that follows a step observes the post-step frame. This makes a recorded NDJSON file fully reproducible — replay produces the same responses regardless of how fast the reader pumps lines.

A `step count:N` produces one response per frame (N total), each carrying the resulting frame number.

### Input injection

Synthetic state replaces MonoGame hardware polling. The injected state persists across frames until explicitly changed — send `"down":false` to clear.

Key names resolve via `Enum.Parse<Keys>()` — use MonoGame's `Keys` enum names verbatim (`Space`, `W`, `Left`, `LeftShift`). Same for gamepad buttons (`Buttons` enum) and axes (`GamepadAxis` enum: `LeftStickX`, `LeftStickY`, `RightStickX`, `RightStickY`, `LeftTrigger`, `RightTrigger`).

Input commands produce no response — query if you need confirmation. `WasKeyPressed` style inputs require the down state to span at least one stepped frame between the down and up commands; combine `input down:true` → `step` → `input down:false` to register a press.

Cursor injection takes screen pixels by default (origin top-left, Y+ down) or world coords with `"space":"world"`. World-space requires at least one registered camera and back-projects through the first one — split-screen disambiguation isn't supported yet, so for those cases use `"space":"screen"` and compute pixels yourself. `primary` and `secondary` mirror left/right mouse buttons; both default to `false` and are sticky across frames until the next cursor command. Once any cursor injection has occurred, the real mouse and touch input are ignored for the rest of the session — there is no opt-out yet.

### Screenshots

`record_next_screenshot` arms a capture that a later `Draw` fulfills — it responds when the PNG is on disk, not when armed. This is how to check rendered output without a human: capture, then read the PNG back. An armed capture holds off both draw suppression and `quit`, so it cannot be starved or exited out from under.

A whole session can go in one write, `quit` included, with a sleep after it to hold stdin open:

```bash
(printf '%s\n' '{"cmd":"step","count":3}' \
   '{"cmd":"record_next_screenshot","path":"shot.png"}' \
   '{"cmd":"step"}' '{"cmd":"quit"}'; sleep 6) \
  | timeout 30 ./MyGame.exe --frb-auto > out.log 2>&1
```

Run the game under `timeout N`, chain the build with `&&` so a failed build cannot fall through to a run, and assert the artifact (`test -f shot.png`) rather than assuming it. Without those, a game that never launched leaves the shell blocked until the tool's own timeout — minutes of wall clock that read as a slow build.

Responses go to stdout and the reader's own log to stderr, so redirect them separately when the log would corrupt your parse.

## Querying Entities (Zero Config)

`query target:"<EntityTypeName>"` returns a snapshot list of every live instance of that type. The type name is the simple C# type name — `PlayerShip`, not `ShmupSpace.Entities.PlayerShip`.

`query target:"entities"` returns every factory keyed by type name, plus any registered providers, in one object.

Snapshots include public instance properties whose type is in a small allowlist: numeric primitives, `bool`, `string`, enums (serialized as their name), `Vector2`, `Vector3`, and `Color` (split to `R/G/B/A`). Reference types like `Sprite`, `Layer`, and the engine itself are skipped — no recursion, no circular-ref blowups. Properties whose getter throws are skipped silently.

A property the agent expects but doesn't see in the snapshot is almost always either non-public, of a non-allowlisted type, or throws on read.

## Setting Values (Zero Config)

`set entity:"Player" prop:"X" value:50.0` looks up the factory for `Player`, takes the first live instance, and assigns via reflection. `value` is always parsed as `double` and converted to the target property's type (including `int`, `float`, enums by ordinal). Errors are specific: missing factory, no live instances, non-existent property, non-writable property.

Boolean/string forcing isn't supported — `value` is numeric only. Register a custom setter if you need that.

## Custom Providers and Setters (Optional)

Use these only for state that *isn't* a plain entity property — typically derived values like `Score`, `Lives`, `current wave`, or a curated projection that hides internals.

```csharp
Engine.RegisterStateProvider("hud", () => new { Score = _score, Lives = _lives });
Engine.RegisterValueSetter("Player", "Health", v => player.Health = (int)v);
```

A registered name takes precedence over reflection — handy when you want a concise view rather than the full property dump.

Providers and setters live on the screen that registered them and disappear on screen exit. The agent can detect a screen change with `query target:"screen"`.

## Gotchas

- **stdout is the protocol channel.** Any `Console.WriteLine` in game code corrupts the NDJSON stream. Use `System.Diagnostics.Debug.WriteLine` for diagnostics — also the project's code-style rule.
- **Type names, not lowercase aliases.** `query target:"Player"` works; `query target:"player"` returns `unknown query target` unless you explicitly registered a `"player"` provider.
- **Screen-scoped providers.** Custom providers registered in `GameScreen.CustomInitialize` don't exist while on `TitleScreen`. Reflection-based entity queries work only for types whose factories have been created — i.e., on the screen that owns them.
- **`quit` calls `Game.Exit()`.** If the game is not yet initialized (e.g. in tests), the call is swallowed silently.