Pokegym Eval

pokegym-eval

PokeGym: A Visually-Driven Long-Horizon Benchmark for Vision-Language Models — Zhang et al. (2026) (arXiv:2604.08340, 2026)

What this evaluates

PokeGym evaluates vision-language models' ability to perform long-horizon planning and spatial reasoning in a complex 3D open-world game using only raw RGB observations. It specifically probes visual grounding, autonomous goal decomposition, and physical deadlock recovery, revealing whether models can navigate cluttered environments, interact with objects, and recover from entrapment without explicit state feedback.

Datasets

• PokeGym — total 30; splits: test (30)

Metrics

• task_completion (primary) — range: [0, 1]
  • Binary outcome per task: 1 if the agent meets the predefined success criteria (e.g., coordinates within bounding box or quest flag triggered) within the fixed step budget, 0 otherwise. Aggregated as the mean success rate across all 30 tasks.

Input / output format

Input: Configurable pure RGB frames (default: current front-view; optional: previous frame, left/right peripheral views), interaction history, and a text prompt specifying the task instruction at one of three granularity levels (Visual-Guided, Step-Guided, or Goal-Only).

Output: Discrete high-level commands (e.g., MoveForward, RotateRight) or parametric control values (maneuver type, execution duration, continuous joystick coordinates X,Y ∈ [-1.0, 1.0]).

Scoring recipe

def compute_task_completion(predictions, gold):
    scores = []
    for task in predictions:
        success = False
        for step, action in enumerate(task.actions):
            env.step(action)
            if gold['verify_success'](env.memory_state):
                success = True
                break
            if step + 1 >= gold['step_budget']:
                break
        scores.append(1 if success else 0)
    return sum(scores) / len(scores)

Common pitfalls

• Step budget exhaustion (180-360 steps) automatically counts as failure, even if the agent is close to the goal.
• Models receive zero internal game state or UI overlays; success relies entirely on interpreting raw RGB frames and inferring spatial relationships.
• Camera rotation is an explicit action required for spatial reasoning, not a passive background feature.

Evidence (verbatim from paper)

Success Criteria: Task completion is threshold-verified using memory variables (e.g., a navigation goal is complete when the coordinates fall within a predefined bounding box). Fixed Step Budget: Each task is assigned a fixed budget of environment steps. Based on heuristic human demonstrations, the budgets range from 180 to 360 environment steps. Termination: An episode terminates under two conditions: (1) Success criteria met; (2) Step budget exhausted.

Citation

@misc{zhang2026pokegym,
  title={PokeGym: A Visually-Driven Long-Horizon Benchmark for Vision-Language Models},
  author={Zhang et al. (2026)},
  year={2026},
  note={arXiv:2604.08340}
}

• arXiv: 2604.08340