Cross Harness

ARC-AGI Cross-Harness

You are managing cross-harness benchmarking for ARC-AGI. This skill lets users generate benchmark instructions for other CLI harnesses (Codex, Gemini CLI, OpenCode), import their results, and compare them against Claude Code runs.

Sub-commands

This skill has three sub-commands, parsed from user arguments after /arc-cross-harness:

If no sub-command is provided, display the table above as usage help and stop.

Supported harness values: codex, gemini, opencode.

Step 1: Pre-flight Checks

1a: Detect Virtual Environment

Determine the venv paths. Do NOT try to source activate scripts.

if [ -f ".arc-agi-venv/bin/python" ]; then
  VENV_PYTHON=".arc-agi-venv/bin/python"
elif [ -f ".arc-agi-venv/Scripts/python.exe" ]; then
  VENV_PYTHON=".arc-agi-venv/Scripts/python.exe"
else
  echo "ERROR: Virtual environment not found. Run /arc-setup first."
  exit 1
fi
echo "VENV_PYTHON=$VENV_PYTHON"

Use $VENV_PYTHON for ALL Python commands below. Store the resolved path.

1b: Read Configuration

$VENV_PYTHON -c "
import json, sys
try:
    with open('.arc-agi-benchmarks/config.json') as f:
        cfg = json.load(f)
    print(json.dumps(cfg, indent=2))
except FileNotFoundError:
    print('ERROR: config.json not found. Run /arc-setup first.', file=sys.stderr)
    sys.exit(1)
"

Extract from the config:

• environments_dir (default: ./environment_files)
• default_seed (default: 0)
• default_max_steps (default: 500)
• default_max_resets (default: 10)

1c: Verify arc-agi is Available

$VENV_PYTHON -c "
from arc_agi import Arcade, OperationMode
from arcengine import GameAction, GameState
print('arc-agi: OK')
"

If this fails, tell the user to run /arc-setup first and stop.

1d: Parse Sub-command and Arguments

Parse the first argument as the sub-command (generate, import, or compare). If not recognized, display usage help and stop.

Then parse remaining arguments based on the sub-command:

For generate:

For import:

For compare:

Validate that harness names are one of: codex, gemini, opencode. If invalid, report:

Invalid harness <NAME>. Supported harnesses: codex, gemini, opencode

Step 2: Execute Sub-command

Branch based on the parsed sub-command. Follow the corresponding section below.

Sub-command: Generate (/arc-cross-harness generate <HARNESS>)

G1: Resolve Game Set and Parameters

If --ref is provided, resolve the reference run:

$VENV_PYTHON -c "
import json, os, sys

ref_arg = '<REF_ARG>'  # 'latest' or a UUID

runs_dir = '.arc-agi-benchmarks/runs'
if ref_arg == 'latest':
    completed = []
    if os.path.isdir(runs_dir):
        for d in os.listdir(runs_dir):
            meta_path = os.path.join(runs_dir, d, 'run-meta.json')
            if os.path.isfile(meta_path):
                with open(meta_path) as f:
                    meta = json.load(f)
                if meta.get('status') == 'completed':
                    completed.append(meta)
    completed.sort(key=lambda m: m.get('timestamp', ''), reverse=True)
    if not completed:
        print(json.dumps({'error': 'No completed runs found. Run /arc-benchmark first.'}))
        sys.exit(1)
    ref_meta = completed[0]
else:
    meta_path = os.path.join(runs_dir, ref_arg, 'run-meta.json')
    if not os.path.isfile(meta_path):
        print(json.dumps({'error': f'Reference run {ref_arg} not found.'}))
        sys.exit(1)
    with open(meta_path) as f:
        ref_meta = json.load(f)

print(json.dumps({
    'run_id': ref_meta.get('run_id'),
    'game_ids': ref_meta.get('game_ids', []),
    'seed': ref_meta.get('seed', 0),
    'max_steps': ref_meta.get('max_steps', 500),
    'max_resets': ref_meta.get('max_resets', 10),
    'game_set': ref_meta.get('game_set', 'all')
}))
"

IMPORTANT: Replace <REF_ARG> with the actual --ref value provided by the user.

If no --ref is provided, resolve the game list from available environments:

$VENV_PYTHON -c "
import json
with open('.arc-agi-benchmarks/config.json') as f:
    cfg = json.load(f)
env_dir = cfg.get('environments_dir', './environment_files')
seed = <SEED>
games_filter = '<GAMES_FILTER>'  # 'all' or comma-separated IDs

from arc_agi import Arcade, OperationMode
op_mode = OperationMode(cfg.get('operation_mode', 'normal'))
arc = Arcade(operation_mode=op_mode, environments_dir=env_dir)
envs = arc.get_environments()

all_ids = []
for env in envs:
    gid = getattr(env, 'game_id', getattr(env, 'id', str(env)))
    all_ids.append(gid)

if games_filter == 'all':
    game_ids = all_ids
else:
    requested = [g.strip() for g in games_filter.split(',')]
    game_ids = [g for g in requested if g in all_ids]
    missing = [g for g in requested if g not in all_ids]
    if missing:
        import sys
        print(f'WARNING: Games not found: {missing}', file=sys.stderr)

print(json.dumps({
    'game_ids': game_ids,
    'seed': seed,
    'max_steps': cfg.get('default_max_steps', 500),
    'max_resets': cfg.get('default_max_resets', 10)
}))
"

IMPORTANT: Replace <SEED> with the seed integer (from --seed or config default) and <GAMES_FILTER> with the --games value (default all).

Store the resolved game_ids, seed, max_steps, max_resets, and optionally the reference run_id.

G2: Generate Instruction Document, Game Driver, and Result Collector

Generate all three files in a single Python script:

$VENV_PYTHON -c "
import json, os, textwrap
from datetime import datetime, timezone

harness = '<HARNESS>'
game_ids = <GAME_IDS_JSON>
seed = <SEED>
max_steps = <MAX_STEPS>
max_resets = <MAX_RESETS>
ref_run_id = '<REF_RUN_ID>'  # or 'none'

with open('.arc-agi-benchmarks/config.json') as f:
    cfg = json.load(f)
env_dir = cfg.get('environments_dir', './environment_files')

out_dir = f'.arc-agi-benchmarks/cross-harness/{harness}'
os.makedirs(f'{out_dir}/runs', exist_ok=True)

# ===== 1. GAME DRIVER SCRIPT =====
game_driver = '''#!/usr/bin/env python3
\"\"\"
ARC-AGI Game Driver for cross-harness benchmarking.
Provides a stdin/stdout interface for any harness to play ARC-AGI games.

Usage:
  python game_driver.py <game_id> <seed> [--env-dir <path>]

Protocol:
  1. On start, prints an observation JSON to stdout.
  2. Reads action JSON from stdin (one line).
  3. Executes action, prints next observation JSON to stdout.
  4. Repeat until state is WIN or GAME_OVER.
  5. On GAME_OVER, send {\"command\": \"reset\"} to reset (up to max_resets).
  6. Send {\"command\": \"quit\"} to end the session.

Observation JSON:
  {
    \"state\": \"PLAYING|WIN|GAME_OVER\",
    \"frame\": [[[int, ...], ...], ...],
    \"levels_completed\": int,
    \"total_levels\": int,
    \"baseline_actions\": [int, ...],
    \"available_actions\": [{\"name\": str, \"is_complex\": bool}, ...],
    \"step\": int,
    \"resets\": int
  }

Action JSON:
  {\"command\": \"step\", \"action\": \"ACTION1\", \"data\": {\"x\": 0, \"y\": 0}, \"reasoning\": \"...\"}
  {\"command\": \"reset\"}
  {\"command\": \"quit\"}
\"\"\"
import json, sys, argparse

def main():
    parser = argparse.ArgumentParser(description='ARC-AGI Game Driver')
    parser.add_argument('game_id', help='Game ID to play')
    parser.add_argument('seed', type=int, help='Random seed')
    parser.add_argument('--env-dir', default='./environment_files', help='Path to environment files')
    parser.add_argument('--max-steps', type=int, default=''' + str(max_steps) + ''', help='Max steps per game')
    parser.add_argument('--max-resets', type=int, default=''' + str(max_resets) + ''', help='Max resets per game')
    args = parser.parse_args()

    from arc_agi import Arcade, OperationMode
    from arcengine import GameAction

    arc = Arcade(operation_mode=OperationMode.NORMAL, environments_dir=args.env_dir)

    # Query environment metadata for baseline_actions and total_levels
    env_meta_baseline_actions = []
    env_meta_total_levels = 5
    try:
        envs = arc.get_environments()
        for e in envs:
            eid = getattr(e, 'game_id', getattr(e, 'id', str(e)))
            if eid == args.game_id:
                env_meta_baseline_actions = getattr(e, 'baseline_actions', [])
                if hasattr(env_meta_baseline_actions, 'tolist'):
                    env_meta_baseline_actions = env_meta_baseline_actions.tolist()
                env_meta_total_levels = getattr(e, 'number_of_levels', getattr(e, 'total_levels', 5))
                break
    except Exception:
        pass

    if env_meta_total_levels == 5 and not env_meta_baseline_actions:
        print(f'WARNING: Could not determine total_levels from environment metadata for {args.game_id}; falling back to total_levels=5', file=sys.stderr, flush=True)

    env = arc.make(args.game_id, seed=args.seed, save_recording=False, render_mode=None)
    # NOTE: In arc-agi, env.observation_space returns a FrameDataRaw object
    # (the current observation), not a Gym-style Space descriptor.
    obs = env.observation_space

    step_count = 0
    reset_count = 0

    def obs_to_dict(obs):
        frame = obs.frame
        if hasattr(frame, 'tolist'):
            frame = frame.tolist()
        actions = []
        for a in env.action_space:
            name = a.name if hasattr(a, 'name') else str(a)
            is_complex = a.is_complex() if hasattr(a, 'is_complex') else False
            actions.append({'name': name, 'is_complex': is_complex})
        return {
            'state': obs.state.name if hasattr(obs.state, 'name') else str(obs.state),
            'frame': frame,
            'levels_completed': getattr(obs, 'levels_completed', 0),
            'total_levels': env_meta_total_levels,
            'baseline_actions': env_meta_baseline_actions,
            'available_actions': actions,
            'step': step_count,
            'resets': reset_count
        }

    # Print initial observation
    print(json.dumps(obs_to_dict(obs)), flush=True)

    for line in sys.stdin:
        line = line.strip()
        if not line:
            continue
        try:
            request = json.loads(line)
        except json.JSONDecodeError:
            print(json.dumps({'error': 'Invalid JSON'}), flush=True)
            continue

        cmd = request.get('command', 'step')

        if cmd == 'quit':
            break
        elif cmd == 'reset':
            if reset_count >= args.max_resets:
                print(json.dumps({'error': 'Max resets exceeded'}), flush=True)
                continue
            obs = env.reset()
            if obs is None:
                print(json.dumps({'error': 'env.reset() returned None'}), flush=True)
                continue
            reset_count += 1
            print(json.dumps(obs_to_dict(obs)), flush=True)
        elif cmd == 'step':
            if step_count >= args.max_steps:
                print(json.dumps({'error': 'Max steps exceeded'}), flush=True)
                continue
            action_name = request.get('action', '')
            try:
                action = GameAction[action_name]
            except KeyError:
                print(json.dumps({'error': f'Unknown action: {action_name}'}), flush=True)
                continue
            step_kwargs = {'action': action}
            if request.get('data'):
                step_kwargs['data'] = request['data']
            if request.get('reasoning'):
                step_kwargs['reasoning'] = {'thought': request['reasoning']}
            obs = env.step(**step_kwargs)
            if obs is None:
                print(json.dumps({'error': 'env.step() returned None'}), flush=True)
                continue
            step_count += 1
            print(json.dumps(obs_to_dict(obs)), flush=True)
        else:
            print(json.dumps({'error': f'Unknown command: {cmd}'}), flush=True)

if __name__ == '__main__':
    main()
'''

with open(f'{out_dir}/game_driver.py', 'w', newline='\\n') as f:
    f.write(game_driver)

# ===== 2. RESULT COLLECTOR SCRIPT =====
collect_results = '''#!/usr/bin/env python3
\"\"\"
ARC-AGI Result Collector for cross-harness benchmarking.
Reads session log files and produces the cross-harness result JSON.

Usage:
  python collect_results.py --sessions-dir <dir> --output <path> --harness <name> [--seed <N>]

Session log format (one file per game, named <game_id>.jsonl):
  Each line is a JSON object with:
    {\"type\": \"observation\", \"data\": {...}}  -- observation from game_driver
    {\"type\": \"action\", \"data\": {...}}       -- action sent to game_driver

If you are using the game_driver.py interactively, you can capture the session
by logging all stdin/stdout exchanges to a JSONL file.

Alternatively, pass --results-file <path> to directly provide a pre-built result JSON
for validation only (the script will validate and pretty-print it).
\"\"\"
import json, sys, argparse, os
from datetime import datetime, timezone

def compute_level_score(baseline_actions, actions_taken, completed):
    if not completed:
        return 0.0
    if actions_taken == 0:
        return 0.0
    return min((baseline_actions / actions_taken) ** 2, 1.0)

def compute_game_score(level_scores):
    if not level_scores:
        return 0.0
    n = len(level_scores)
    weighted = sum(level_scores[i] * (i + 1) for i in range(n))
    total_weight = sum(range(1, n + 1))
    return weighted / total_weight if total_weight > 0 else 0.0

def main():
    parser = argparse.ArgumentParser(description='ARC-AGI Result Collector')
    parser.add_argument('--sessions-dir', help='Directory containing session JSONL files')
    parser.add_argument('--results-file', help='Pre-built result JSON to validate')
    parser.add_argument('--output', default='result.json', help='Output file path')
    parser.add_argument('--harness', default=''' + repr(harness) + ''', help='Harness name')
    parser.add_argument('--seed', type=int, default=''' + str(seed) + ''', help='Seed used')
    parser.add_argument('--model', default='unknown', help='Model identifier')
    parser.add_argument('--version', default='unknown', help='Harness version')
    parser.add_argument('--notes', default='', help='Additional notes')
    args = parser.parse_args()

    if args.results_file:
        with open(args.results_file) as f:
            result = json.load(f)
        print(json.dumps(result, indent=2))
        return

    if not args.sessions_dir:
        print('ERROR: --sessions-dir or --results-file required', file=sys.stderr)
        sys.exit(1)

    games = []
    for fname in sorted(os.listdir(args.sessions_dir)):
        if not fname.endswith('.jsonl'):
            continue
        game_id = fname.replace('.jsonl', '')
        observations = []
        actions = []
        with open(os.path.join(args.sessions_dir, fname)) as f:
            for line in f:
                line = line.strip()
                if not line:
                    continue
                entry = json.loads(line)
                if entry.get('type') == 'observation':
                    observations.append(entry['data'])
                elif entry.get('type') == 'action':
                    actions.append(entry['data'])

        if not observations:
            games.append({
                'game_id': game_id,
                'state': 'NOT_PLAYED',
                'levels_completed': 0,
                'total_levels': 5,
                'total_actions': 0,
                'total_resets': 0,
                'levels': []
            })
            continue

        last_obs = observations[-1]
        first_obs = observations[0]
        state = last_obs.get('state', 'GAME_OVER')
        levels_completed = last_obs.get('levels_completed', 0)
        total_actions = sum(1 for a in actions if a.get('command') == 'step')
        total_resets = sum(1 for a in actions if a.get('command') == 'reset')

        # Read baseline_actions and total_levels from observation data (set by game_driver.py)
        env_baseline_actions = first_obs.get('baseline_actions', [])
        env_total_levels = first_obs.get('total_levels', 5)

        # Build level info by iterating action entries with command='step'.
        # We pair each step action with the observation that follows it
        # (observations[action_index + 1]) to detect level transitions.
        # This avoids over-counting from initial/reset observations.
        levels = []
        current_level = 0
        level_action_count = 0
        step_obs_idx = 0  # tracks which observation corresponds to next step result
        for act in actions:
            if act.get('command') != 'step':
                continue
            step_obs_idx += 1  # skip past the preceding observation
            level_action_count += 1
            # The observation after this step (if available) tells us the new state
            if step_obs_idx < len(observations):
                lc = observations[step_obs_idx].get('levels_completed', 0)
                if lc > current_level:
                    ba = env_baseline_actions[current_level] if current_level < len(env_baseline_actions) else 0
                    levels.append({
                        'level_index': current_level + 1,
                        'completed': True,
                        'actions_taken': level_action_count,
                        'baseline_actions': ba
                    })
                    current_level = lc
                    level_action_count = 0

        # Add final level (may be incomplete)
        if level_action_count > 0 or not levels:
            ba = env_baseline_actions[current_level] if current_level < len(env_baseline_actions) else 0
            levels.append({
                'level_index': current_level + 1,
                'completed': state == 'WIN' and current_level + 1 <= levels_completed,
                'actions_taken': level_action_count,
                'baseline_actions': ba
            })

        games.append({
            'game_id': game_id,
            'state': state,
            'levels_completed': levels_completed,
            'total_levels': env_total_levels,
            'total_actions': total_actions,
            'total_resets': total_resets,
            'levels': levels
        })

    result = {
        'schema_version': '1.0.0',
        'scoring_formula_version': '1.0.0',
        'harness': args.harness,
        'timestamp': datetime.now(timezone.utc).isoformat(),
        'seed': args.seed,
        'games': games,
        'metadata': {
            'model': args.model,
            'version': args.version,
            'notes': args.notes
        }
    }

    with open(args.output, 'w') as f:
        json.dump(result, f, indent=2)
    print(f'Result written to {args.output}')
    print(f'Games: {len(games)}, Completed: {sum(1 for g in games if g[\"state\"] == \"WIN\")}')

if __name__ == '__main__':
    main()
'''

with open(f'{out_dir}/collect_results.py', 'w', newline='\\n') as f:
    f.write(collect_results)

# ===== 3. INSTRUCTION DOCUMENT =====

game_ids_str = ', '.join(game_ids)
game_ids_json = json.dumps(game_ids)
ref_line = f'Reference run: {ref_run_id}' if ref_run_id != 'none' else 'Reference run: none'
now = datetime.now(timezone.utc).isoformat()

# Harness-specific sections
if harness == 'codex':
    harness_guidance = '''## Harness-Specific Guidance: Codex CLI

### Sandbox Awareness

Codex operates in a sandboxed environment. You must install `arc-agi` inside the sandbox:

```bash
pip install arc-agi

Verify the installation:

python -c \"from arc_agi import Arcade, OperationMode; print('OK')\"

Execution Model

Codex uses file I/O rather than interactive stdin/stdout. The recommended approach:

1. Write a Python script for each game (or a single script that iterates all games)
2. The script should:
   • Initialize the game via the arc-agi API
   • Analyze the frame data
   • Choose actions based on pattern recognition
   • Loop until WIN or GAME_OVER
   • Write results to a JSON file
3. Execute the script via Codex sandbox

Running Codex

Use the codex exec command for unattended execution:

codex exec -m <model> --full-auto "<prompt>"

Additional flags:

• --sandbox workspace-write -- control sandbox write permissions
• --json -- enable JSON event streaming for structured output parsing

Prompt Template

Use this system prompt when invoking Codex:

You are playing ARC-AGI games. For each game, you will analyze grid patterns and choose actions to transform the grid. Use the game_driver.py script to interact with games. Read observations, reason about patterns, and submit actions. Collect results using collect_results.py.

Limitations

• Codex sandbox may have network restrictions -- ensure arc-agi is pre-installed
• File system access is sandboxed -- keep all files in the working directory ''' elif harness == 'gemini': harness_guidance = '''## Harness-Specific Guidance: Gemini CLI

Tool Use

Gemini CLI uses function calling / tool_use for shell commands. Ensure tool_use is enabled so Gemini can execute Python commands.

Running Gemini CLI

For headless (non-interactive) mode, use the -p flag:

gemini -p "<prompt>"

Use --output-format json for structured output parsing:

gemini -p "<prompt>" --output-format json

Important: Gemini CLI has a 5-minute timeout on shell tool invocations. For games that may take longer, consider breaking them into smaller batches or using a wrapper script that handles individual game sessions.

Execution Model

Gemini drives a Python session, reading observations and choosing actions:

1. Start by running the game_driver.py script
2. Gemini reads the observation JSON output
3. Gemini reasons about the grid pattern
4. Gemini sends an action JSON to stdin
5. Repeat until WIN or GAME_OVER

Prompt Structure

Structure the Gemini conversation to maintain context across turns within a game:

You are playing ARC-AGI games. Use the provided game_driver.py to interact with each game. Run it as: python game_driver.py Read the JSON observations, analyze the grid patterns, and send action JSON via stdin. After all games, run collect_results.py to produce the final result file.

Multi-Turn Context

Gemini should maintain context across turns within a game. Each observation builds on the previous one. Share your reasoning about the pattern as you discover it across levels. ''' elif harness == 'opencode': harness_guidance = '''## Harness-Specific Guidance: OpenCode

Configuration

Create or update opencode.json (no leading dot) in the project root:

{
  \"tools\": {
    \"shell\": true
  }
}

Running OpenCode

For scripted execution, use the opencode run command:

opencode run --model <model> "<prompt>"

Backend Flexibility

OpenCode supports multiple LLM backends. Configure your preferred backend in opencode.json. The game interaction is the same regardless of backend.

Execution Model

Similar to Claude Code -- OpenCode uses shell tool execution to run Python scripts:

1. Execute python game_driver.py <game_id> <seed> via shell tool
2. Read the observation JSON
3. Reason about the grid pattern
4. Send action JSON to stdin (or use the batch approach below)
5. Repeat until WIN or GAME_OVER

Session Management

Maintain game state across OpenCode tool calls. The game_driver.py uses stdin/stdout, so you can either:

• Run it interactively in a single shell session
• Use the batch approach: write a wrapper script that plays the full game autonomously ''' else: harness_guidance = ''

instructions = f'''# ARC-AGI Cross-Harness Benchmark Instructions

Target harness: {harness} Generated: {now} {ref_line} Game set: {len(game_ids)} games Seed: {seed} Max steps per game: {max_steps} Max resets per game: {max_resets}

Prerequisites

• Python >= 3.12
• pip or uv package manager

Environment Setup

1. Install arc-agi

pip install arc-agi

Or with uv:

uv add arc-agi

2. Verify Installation

python -c \"from arc_agi import Arcade, OperationMode; print('OK')\"

3. Environment Files

Ensure ARC-AGI environment files are available. They should be in {env_dir} relative to the project root, or set a custom path when running the game driver:

python game_driver.py <game_id> {seed} --env-dir {env_dir}

Game Set

Play these {len(game_ids)} games with seed={seed}:

{game_ids_str}

Game IDs as JSON array (for scripting):

{game_ids_json}

Game Interaction Protocol

How ARC-AGI Games Work

Each game has multiple levels (typically 5). Each level presents a grid (2D array of integers). You must figure out the transformation rule and apply it by choosing actions. The same rule applies across all levels -- use early levels to learn the pattern.

Grid Interpretation

• frame is a list of 2D integer arrays (grid layers)
• Each integer represents a color (0 = background, 1-9 = colors)
• Grid sizes start small (e.g., 3x3) and grow with each level
• Look for: symmetry, borders, objects, repeating patterns, color relationships

Using the Game Driver

The game_driver.py script (included alongside these instructions) provides a simple stdin/stdout protocol:

python game_driver.py <game_id> {seed} --env-dir {env_dir}

Observation output (printed to stdout as JSON):

{{
  "state": "PLAYING",
  "frame": [[[0, 1, 0], [1, 0, 1], [0, 1, 0]]],
  "levels_completed": 0,
  "total_levels": 5,
  "baseline_actions": [4, 6, 8, 10, 12],
  "available_actions": [
    {{"name": "ACTION1", "is_complex": false}},
    {{"name": "ACTION2", "is_complex": true}}
  ],
  "step": 0,
  "resets": 0
}}

Action input (send as JSON to stdin):

For simple actions (is_complex=false):

{{"command": "step", "action": "ACTION1", "reasoning": "Applying rotation pattern"}}

For complex actions (is_complex=true, requiring x,y coordinates):

{{"command": "step", "action": "ACTION2", "data": {{"x": 2, "y": 1}}, "reasoning": "Filling cell at column 2, row 1"}}

To reset after GAME_OVER:

{{"command": "reset"}}

To end the session:

{{"command": "quit"}}

Game States

Action Space

Important: The set of available actions varies between games and may change between levels within the same game. Always consult the available_actions array in each observation before choosing an action. Do not assume actions from one game are valid in another.

Strategy Tips

1. Level 1: Experiment to discover what each action does
2. Level 2+: Apply the rule learned from earlier levels
3. After reset: Try a different approach based on what you learned
4. Efficiency matters: Fewer actions = higher score

{harness_guidance}

Scoring Reference

Scores are calculated using these formulas:

Per-level score (0.0 to 1.0):

level_score = min((baseline_actions / actions_taken) ** 2, 1.0) if completed else 0.0

Per-game score (weighted average, later levels weighted more):

game_score = sum(level_score[i] * (i+1) for i in 0..N-1) / sum(1..N)

Overall score:

overall_score = average(all game scores)

Result Collection

After playing all games, produce a result JSON file using collect_results.py or manually.

Automated Collection

If you logged sessions as JSONL files (one per game, in a sessions/ directory):

python collect_results.py \\
  --sessions-dir ./sessions \\
  --output result.json \\
  --harness {harness} \\
  --seed {seed} \\
  --model \"<your-model-name>\" \\
  --version \"<harness-version>\" \\
  --notes \"<any context>\"

Manual Result File

Create a JSON file matching this schema:

{{
  "schema_version": "1.0.0",
  "scoring_formula_version": "1.0.0",
  "harness": "{harness}",
  "timestamp": "<ISO 8601 timestamp>",
  "seed": {seed},
  "games": [
    {{
      "game_id": "<game_id>",
      "state": "WIN or GAME_OVER or NOT_PLAYED",
      "levels_completed": 5,
      "total_levels": 5,
      "total_actions": 45,
      "total_resets": 0,
      "levels": [
        {{
          "level_index": 1,
          "completed": true,
          "actions_taken": 4,
          "baseline_actions": 4
        }}
      ]
    }}
  ],
  "metadata": {{
    "model": "<model used>",
    "version": "<harness version>",
    "notes": "<any context>"
  }}
}}

Result Template

Here is a pre-filled template with all {len(game_ids)} game IDs:

import json
from datetime import datetime, timezone

games = []
for gid in {game_ids_json}:
    games.append({{
        \"game_id\": gid,
        \"state\": \"NOT_PLAYED\",
        \"levels_completed\": 0,
        \"total_levels\": 5,
        \"total_actions\": 0,
        \"total_resets\": 0,
        \"levels\": []
    }})

result = {{
    \"schema_version\": \"1.0.0\",
    \"scoring_formula_version\": \"1.0.0\",
    \"harness\": \"{harness}\",
    \"timestamp\": datetime.now(timezone.utc).isoformat(),
    \"seed\": {seed},
    \"games\": games,
    \"metadata\": {{
        \"model\": \"<your-model>\",
        \"version\": \"<version>\",
        \"notes\": \"\"
    }}
}}

with open(\"result.json\", \"w\") as f:
    json.dump(result, f, indent=2)
print(\"Template written to result.json -- fill in game results.\")

Importing Results

Once you have the result JSON file, import it back into the ARC-AGI benchmarker:

/arc-cross-harness import {harness} <path-to-result.json>

This will validate, normalize scores, and store the results for comparison. '''

with open(f'{out_dir}/instructions.md', 'w', newline='\n') as f: f.write(instructions)

print(json.dumps({ 'status': 'generated', 'instructions': f'{out_dir}/instructions.md', 'game_driver': f'{out_dir}/game_driver.py', 'collect_results': f'{out_dir}/collect_results.py', 'game_count': len(game_ids), 'seed': seed, 'ref_run_id': ref_run_id })) "

**IMPORTANT**: Replace the following angle-bracket placeholders with actual literal values:
- `<HARNESS>`: the harness name (e.g., `codex`)
- `<GAME_IDS_JSON>`: Python list literal of game IDs (e.g., `["bt11", "ls20"]`)
- `<SEED>`: integer seed value
- `<MAX_STEPS>`: integer max steps
- `<MAX_RESETS>`: integer max resets
- `<REF_RUN_ID>`: the reference run ID string, or `none` if no `--ref` was provided

### G3: Report to User

After generation, report:

Instructions generated: .arc-agi-benchmarks/cross-harness//instructions.md Game driver: .arc-agi-benchmarks/cross-harness//game_driver.py Result collector: .arc-agi-benchmarks/cross-harness//collect_results.py Game set: games, seed= Reference run: <RUN_ID or "none">

Next steps:

1. Open the instructions file and follow the setup steps in the target harness
2. Run the benchmark on the target harness
3. Save the result JSON file
4. Import with: /arc-cross-harness import

---

## Sub-command: Import (`/arc-cross-harness import <HARNESS> <PATH>`)

### I1: Validate the Result File

```bash
$VENV_PYTHON -c "
import json, sys, os

harness = '<HARNESS>'
result_path = '<RESULT_PATH>'

# Validate harness
if harness not in ('codex', 'gemini', 'opencode'):
    print(f'ERROR: Invalid harness \"{harness}\". Supported: codex, gemini, opencode', file=sys.stderr)
    sys.exit(1)

# Validate file exists
if not os.path.isfile(result_path):
    print(f'ERROR: Result file not found: {result_path}', file=sys.stderr)
    sys.exit(1)

# Read and validate schema
with open(result_path) as f:
    try:
        result = json.load(f)
    except json.JSONDecodeError as e:
        print(f'ERROR: Invalid JSON in result file: {e}', file=sys.stderr)
        sys.exit(1)

errors = []
warnings = []

# Required top-level fields
for field in ['schema_version', 'harness', 'timestamp', 'seed', 'games', 'metadata']:
    if field not in result:
        errors.append(f'Missing required field: {field}')

if 'scoring_formula_version' not in result:
    result['scoring_formula_version'] = '1.0.0'

# Warn on harness mismatch
if result.get('harness') and result['harness'] != harness:
    warnings.append(f'Harness field in result (\"{result[\"harness\"]}\") does not match argument (\"{harness}\")')

# Warn on schema version
if result.get('schema_version') and result['schema_version'] != '1.0.0':
    warnings.append(f'Schema version {result[\"schema_version\"]} may not be fully supported (expected 1.0.0)')

# Validate games array
games = result.get('games', [])
if not isinstance(games, list):
    errors.append('\"games\" must be an array')
else:
    for i, game in enumerate(games):
        prefix = f'games[{i}]'
        for field in ['game_id', 'state', 'levels_completed', 'total_levels', 'total_actions', 'total_resets', 'levels']:
            if field not in game:
                errors.append(f'Missing required field: {prefix}.{field}')

        levels = game.get('levels', [])
        if isinstance(levels, list):
            for j, level in enumerate(levels):
                lprefix = f'{prefix}.levels[{j}]'
                for field in ['level_index', 'completed', 'actions_taken', 'baseline_actions']:
                    if field not in level:
                        errors.append(f'Missing required field: {lprefix}.{field}')

if errors:
    print('Schema validation errors:', file=sys.stderr)
    for e in errors:
        print(f'  - {e}', file=sys.stderr)
    sys.exit(1)

if warnings:
    for w in warnings:
        print(f'WARNING: {w}', file=sys.stderr)

print(json.dumps({'status': 'valid', 'games': len(games), 'warnings': warnings}))
"

IMPORTANT: Replace <HARNESS> with the harness name and <RESULT_PATH> with the actual file path to the result JSON.

I2: Normalize Scores and Create Run Directory

$VENV_PYTHON -c "
import json, sys, os, uuid
from datetime import datetime, timezone

harness = '<HARNESS>'
result_path = '<RESULT_PATH>'

with open(result_path) as f:
    result = json.load(f)

if 'scoring_formula_version' not in result:
    result['scoring_formula_version'] = '1.0.0'

# Duplicate import check: reject if same timestamp+harness already imported
import_ts = result.get('timestamp', '')
existing_runs_dir = f'.arc-agi-benchmarks/cross-harness/{harness}/runs'
if os.path.isdir(existing_runs_dir):
    for existing_dir in os.listdir(existing_runs_dir):
        existing_meta_path = os.path.join(existing_runs_dir, existing_dir, 'run-meta.json')
        if os.path.isfile(existing_meta_path):
            with open(existing_meta_path) as ef:
                existing_meta = json.load(ef)
            if existing_meta.get('timestamp') == import_ts and existing_meta.get('harness') == harness:
                print(f'ERROR: Duplicate import detected. A run with timestamp {import_ts} for harness {harness} already exists (run_id: {existing_dir}).', file=sys.stderr)
                sys.exit(1)

run_id = str(uuid.uuid4())
run_dir = f'.arc-agi-benchmarks/cross-harness/{harness}/runs/{run_id}'
os.makedirs(run_dir, exist_ok=True)

games = result.get('games', [])
metadata = result.get('metadata', {})

# ===== Recalculate all scores =====

def compute_level_score(baseline_actions, actions_taken, completed):
    if not completed:
        return 0.0
    if actions_taken <= 0:
        return 0.0
    return min((baseline_actions / actions_taken) ** 2, 1.0)

def compute_game_score(level_scores):
    if not level_scores:
        return 0.0
    n = len(level_scores)
    weighted = sum(level_scores[i] * (i + 1) for i in range(n))
    total_weight = sum(range(1, n + 1))
    return weighted / total_weight if total_weight > 0 else 0.0

total_environments = len(games)
total_environments_completed = 0
total_levels_completed_sum = 0
total_levels_sum = 0
total_actions_sum = 0
game_scores = []

scorecard_games = []

for game in games:
    game_id = game['game_id']
    state = game.get('state', 'NOT_PLAYED')
    levels_completed = game.get('levels_completed', 0)
    total_levels = game.get('total_levels', 5)
    total_actions = game.get('total_actions', 0)
    total_resets = game.get('total_resets', 0)
    levels = game.get('levels', [])

    if state == 'WIN':
        total_environments_completed += 1
    total_levels_completed_sum += levels_completed
    total_levels_sum += total_levels
    total_actions_sum += total_actions

    # Compute per-level scores
    level_scores_list = []
    level_actions_list = []
    level_baselines_list = []
    for lvl in levels:
        ls = compute_level_score(
            lvl.get('baseline_actions', 0),
            lvl.get('actions_taken', 0),
            lvl.get('completed', False)
        )
        level_scores_list.append(ls)
        level_actions_list.append(lvl.get('actions_taken', 0))
        level_baselines_list.append(lvl.get('baseline_actions', 0))

    # Pad to total_levels if needed
    while len(level_scores_list) < total_levels:
        level_scores_list.append(0.0)
        level_actions_list.append(0)
        level_baselines_list.append(0)

    game_score = compute_game_score(level_scores_list)
    game_scores.append(game_score)

    run_guid = str(uuid.uuid4())
    scorecard_games.append({
        'id': game_id,
        'score': game_score,
        'runs': [{
            'id': game_id,
            'guid': run_guid,
            'score': game_score,
            'levels_completed': levels_completed,
            'actions': total_actions,
            'resets': total_resets,
            'state': state,
            'completed': state == 'WIN',
            'level_scores': level_scores_list,
            'level_actions': level_actions_list,
            'level_baseline_actions': level_baselines_list,
            'number_of_levels': total_levels
        }]
    })

overall_score = sum(game_scores) / len(game_scores) if game_scores else 0.0

# ===== Write run-meta.json =====

run_meta = {
    'run_id': run_id,
    'harness': harness,
    'timestamp': result.get('timestamp', datetime.now(timezone.utc).isoformat()),
    'duration_seconds': 0,
    'game_set': 'imported',
    'game_ids': [g['game_id'] for g in games],
 

…(truncated)