Rmbench Eval

rmbench-eval

RMBench: Memory-Dependent Robotic Manipulation Benchmark with Insights into Policy Design — Chen et al. (2026) (arXiv:2603.01229, 2026)

What this evaluates

This benchmark evaluates robotic manipulation policies on memory-dependent, non-Markovian tasks. It probes a model's ability to retain and utilize historical visual and state information over long horizons to complete multi-step dual-arm manipulation sequences.

Datasets

• RMBench — total ?; splits: test (-1); repo https://github.com/robotwin-Platform/rmbench

Metrics

• success rate (primary) — range: percent
  • Calculated as the number of successfully completed rollout episodes divided by the total number of evaluated episodes, multiplied by 100 to yield a percentage.

Input / output format

Input: Visual observations (image tokens) and task instructions, optionally augmented with memory tokens (anchor, sliding window, or key memory) from previous timesteps.

Output: Next robotic action (e.g., joint commands or end-effector pose).

Scoring recipe

def compute_success_rate(predictions, gold, total_episodes):
    successes = 0
    for i in range(total_episodes):
        if predictions[i] == gold[i]:  # Binary flag: 1=success, 0=failure
            successes += 1
    return (successes / total_episodes) * 100

Common pitfalls

• Baseline models trained under Markovian assumptions fail on non-Markovian tasks because they only use current observations, ignoring necessary historical context.
• Inaccurate subtask termination detection by the classifier can cause repeated actions or missed contacts, artificially driving success rates to zero on specific tasks (e.g., Press Button).
• Fixed observation windows or sliding memory without anchor retention lead to progressive loss of task-critical information over long horizons.

Evidence (verbatim from paper)

On average, Mem-0 improves success rates by 38.4% on $M(1)$ tasks and 21.2% on $M(n)$ tasks relative to the baselines, underscoring the critical role of memory modules in addressing memory-dependent manipulation in RMBench.

Citation

@misc{chen2026rmbench,
  title={RMBench: Memory-Dependent Robotic Manipulation Benchmark with Insights into Policy Design},
  author={Chen et al. (2026)},
  year={2026},
  note={arXiv:2603.01229}
}

• arXiv: 2603.01229