Competitive Pokemon Singles Eval

competitive-pokemon-singles-eval

Human-Level Competitive Pok'emon via Scalable Offline Reinforcement Learning with Transformers — Grigsby et al. (2025) (arXiv:2504.04395, 2025)

What this evaluates

Evaluates an agent's ability to play competitive Pokémon Singles under partial observability and long-horizon uncertainty. It measures strategic decision-making, team building, and adaptation against heuristic opponents, search-based engines, LLM agents, and human players on a ranked ladder.

Datasets

• Competitive Pokémon Singles (CPS) on Pokémon Showdown — total ?; splits: Variety Set (1000), Replay Set (-1), Competitive Set (-1)

Metrics

• win rate (primary) — range: percent
  • Calculated as the number of battles won divided by the total number of battles played against a specific opponent or set of opponents.
• Glicko-1 / GXE — range: rating / percent
  • Glicko-1 rating system and Glicko-1 Expected Win rate (GXE) computed by Pokémon Showdown based on match outcomes against human players on the ranked ladder.

Input / output format

Input: First-person battle state representation including current team, opponent's revealed Pokémon, move history, and turn-by-turn actions. Teams are provided as prompts from the Variety, Replay, or Competitive sets.

Output: A single action per turn: choose a move, switch a Pokémon, or use an item/field effect, given the current game state.

Scoring recipe

def compute_win_rate(battles):
    wins = sum(1 for b in battles if b.result == 'win')
    return wins / len(battles)

def compute_glicko_gxe(battles):
    ratings = [b.glicko1 for b in battles]
    gxes = [b.gxe for b in battles]
    return sum(ratings) / len(ratings), sum(gxes) / len(gxes)

Common pitfalls

• Win rates are highly sensitive to the team set used for prompting (Variety vs. Replay vs. Competitive), making cross-experiment comparisons difficult without standardizing the set.
• Ladder-based metrics like Glicko-1 and GXE are noisy and can be artificially deflated during tournaments when top players create new accounts, requiring careful percentile estimation.
• Partial observability requires inferring opponent teams from battle logs; imperfect replay reconstruction can lead to inconsistent evaluation conditions across generations.

Evidence (verbatim from paper)

Win rates are measured over large samples of hundreds or thousands of battles unless otherwise noted. The average win rate against 6 of these heuristics on the Variety Set forms a “Heuristic Composite Score”. Models’ Glicko-1 and GXE stats at the end of their final battle are shown in Figure [12].

Citation

@misc{grigsby2025competitivepokemon,
  title={Human-Level Competitive Pok'emon via Scalable Offline Reinforcement Learning with Transformers},
  author={Grigsby et al. (2025)},
  year={2025},
  note={arXiv:2504.04395}
}

• arXiv: 2504.04395