Memoryvla Eval

memoryvla-eval

MemoryVLA: Perceptual-Cognitive Memory in Vision-Language-Action Models for Robotic Manipulation — Shi et al. (2025) (arXiv:2508.19236, 2025)

What this evaluates

Evaluates long-horizon robotic manipulation capabilities of vision-language-action models under non-Markovian dynamics. It probes the model's ability to maintain and retrieve perceptual and semantic memory over extended task horizons using only third-person visual observations and language instructions.

Datasets

• SimplerEnv-Bridge — total ?; splits: test (-1)
• SimplerEnv-Fractal — total ?; splits: test (-1)
• LIBERO — total ?; splits: test (-1)
• Real-world Manipulation — total ?; splits: test (-1)

Metrics

• success rate (primary) — range: percent
  • Percentage of successful task executions out of total trials per task. Calculated as (number of successful trials / total trials) × 100.

Input / output format

Input: Single third-person RGB frame at 224×224 together with the language instruction.

Output: 7-DoF actions.

Scoring recipe

def compute_success_rate(trials, success_condition):
    successes = sum(1 for trial in trials if success_condition(trial))
    return (successes / len(trials)) * 100

Common pitfalls

• Baselines like π₀ and CogACT often use additional proprioceptive states or wrist-camera views; MemoryVLA strictly uses only third-person RGB and language, making direct comparison sensitive to input modality assumptions.
• Real-world long-horizon tasks use step-wise scoring to track progress over sub-goals rather than binary end-state success, which can inflate or deflate scores depending on the evaluation script.
• SimplerEnv-Fractal's Visual Aggregation (VA) setting intentionally alters backgrounds, lighting, and distractors to stress-test robustness, differing from standard visual matching protocols.

Evidence (verbatim from paper)

Results are reported at the best validation step, and each task is evaluated with 24 trials to compute success rates.

Citation

@misc{shi2025memoryvla,
  title={MemoryVLA: Perceptual-Cognitive Memory in Vision-Language-Action Models for Robotic Manipulation},
  author={Shi et al. (2025)},
  year={2025},
  note={arXiv:2508.19236}
}

• arXiv: 2508.19236