Gym V Eval

gym-v-eval

Gym-V: A Unified Vision Environment System for Agentic Vision Research — Meng et al. (2026) (arXiv:2603.15432, 2026)

What this evaluates

Evaluates agentic vision models on zero-shot generalization across 179 procedurally generated environments spanning 10 domains. It measures task completion via answer correctness for single-turn interactions and cumulative performance via normalized episodic return for multi-turn interactions.

Datasets

• Gym-V — total 179; splits: test (-1); repo https://github.com/ModalMinds/gym-v

Metrics

• answer correctness — range: [0, 100] percent
  • Binary score (1 or 0) indicating whether the model's generated answer exactly matches the ground truth for single-turn environments.
• normalized episodic return (primary) — range: [0, 100] percent
  • Sum of step-wise rewards over an episode, normalized, with any negative values explicitly clipped to zero. Used for multi-turn environments.

Input / output format

Input: Visual observations from procedurally generated environments across 10 domains. Models interact via single-turn (one observation to answer) or multi-turn (sequential observations to actions) protocols.

Output: Single-turn: a final answer string. Multi-turn: a sequence of discrete actions over an episode.

Scoring recipe

def score(prediction, gold, env_type, rewards=None):
    if env_type == 'single-turn':
        return 1.0 if prediction == gold else 0.0
    else:  # multi-turn
        raw_return = sum(rewards) if rewards else 0.0
        return max(0.0, raw_return)  # negative values clipped to zero

Common pitfalls

• Failing to distinguish between single-turn (answer correctness) and multi-turn (episodic return) scoring protocols, which use fundamentally different metrics.
• Overlooking that negative episodic returns are explicitly clipped to zero before being converted to percentages, which can mask poor performance in multi-turn tasks.
• Assuming 'mean@3' refers to top-3 generation; in this context it denotes averaging scores over three independent runs or seeds per environment.

Evidence (verbatim from paper)

Single-turn environments are scored by answer correctness, while multi-turn environments report normalized episodic return (negative values clipped to zero). All scores are shown as percentages ($\times 100$).

Citation

@misc{meng2026gymv,
  title={Gym-V: A Unified Vision Environment System for Agentic Vision Research},
  author={Meng et al. (2026)},
  year={2026},
  note={arXiv:2603.15432}
}

• arXiv: 2603.15432