Wiser Eval

wiser-eval

Grounded World Model for Semantically Generalizable Planning — Quanyi Li et al. (2026) (arXiv:2604.11751, 2026)

What this evaluates

Evaluates a robot's ability to generalize visuomotor planning to unseen visual signals, referring expressions, and spatial relationships during cube-picking and placing tasks. It probes semantic generalization and robustness to distribution shifts in embodied instruction following.

Datasets

• WISER Benchmark — total ?; splits: train (-1), test (288); repo https://github.com/QuanyiLi/gwm-wiser

Metrics

• Success (primary) — range: [0, 1]
  • Fraction of tasks where the robot successfully executes the instructed manipulation (grasping the correct object and placing it at the target location). Computed as successful tasks divided by total tasks.

Input / output format

Input: Natural language task instructions (system prompt + task prompt), current robot observations (images from wrist and other cameras), and robot state.

Output: Action sequences or trajectories for robot execution, typically proposed via MPC and scored via cosine similarity to instruction embeddings.

Scoring recipe

def compute_success(predictions, gold):
    correct = 0
    for pred, gold_inst in zip(predictions, gold):
        if check_grasp_correctness(pred, gold_inst) and check_place_correctness(pred, gold_inst):
            correct += 1
    return correct / len(gold)

Common pitfalls

• Models often overfit to training sentence structures and lose the ability to understand decomposed instructions.
• Performance is bottlenecked by the scoring model (Qwen3-VL-Embedding); even ground-truth future embeddings do not yield 100% success.
• Delta vs. absolute action spaces do not inherently improve generalizability, contrary to some assumptions.

Evidence (verbatim from paper)

The GWM-MPC achieves the best test-scene performance, yielding an 87% success rate across 288 test tasks that feature unseen referring expressions, spatial relationship descriptions, and visual signals.

Citation

@misc{li2026groundedworldmodel,
  title={Grounded World Model for Semantically Generalizable Planning},
  author={Quanyi Li et al. (2026)},
  year={2026},
  note={arXiv:2604.11751}
}

• arXiv: 2604.11751