Embodiedcomp Eval

embodiedcomp-eval

Embodied Image Compression — Li et al. (2025) (arXiv:2512.11612, 2025)

What this evaluates

Evaluates how image compression codecs impact the performance of Vision-Language-Action (VLA) models in closed-loop robotic manipulation tasks under ultra-low bitrates. It measures whether compressed visual inputs cause task failure or require excessive inference steps, highlighting the disconnect between traditional visual fidelity metrics and embodied AI operational requirements.

Datasets

• EmbodiedComp — total ?; splits: train (2000), test (100)

Metrics

• Success Rate (SR) (primary) — range: [0, 1]
  • The fraction of test scenes in which the VLA successfully executes the commanded primitive action (pick, push, or press) within the maximum iteration step budget.
• Step — range: other
  • The number of VLA inference iterations required to complete the task or reach the maximum step budget. Exceeding 150 steps is treated as task failure.

Input / output format

Input: Compressed/distorted 256x256 images rendered from a Robosuite/MuJoCo simulation (third-person or first-person gripper view), paired with natural language commands referring to a single main object.

Output: Discrete action commands (pick, push, or press) predicted by the VLA, which are executed in the simulator to update the environment state.

Scoring recipe

success_count = 0
steps_list = []
for scene in test_scenes:
    state = scene.initial_state
    for step in range(1, max_steps + 1):
        img = render(state)
        img_compressed = compress(img, target_bpp)
        action = vla.predict(img_compressed, command)
        state = step_simulation(state, action)
        if is_success(state, command):
            success_count += 1
            steps_list.append(step)
            break
    else:
        steps_list.append(max_steps)
sr = success_count / len(test_scenes)
avg_steps = sum(steps_list) / len(test_scenes)

Common pitfalls

• Using traditional image fidelity metrics (e.g., PSNR, SSIM) instead of task-based metrics, as the paper demonstrates a fundamental disconnect between CV-based fidelity and embodied agent performance.
• Evaluating zero-shot VLAs directly, which the protocol explicitly forbids; models must be fine-tuned on the train split first to isolate compression distortion from policy limitations.
• Misinterpreting the Step metric: exceeding 150 iterations is classified as failure, not merely a high cost, due to the positive-feedback degradation regime.

Evidence (verbatim from paper)

Consequently, EmbodiedComp adopts only two indicators: (i) Success Rate (SR): the fraction of scenes in which the command is ultimately satisfied. (ii) Step: the number of VLA iterations to reach success or exhaust the budget.

Citation

@misc{li2025embodied,
  title={Embodied Image Compression},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2512.11612}
}

• arXiv: 2512.11612