Marketgen Eval

marketgen-eval

MarketGen: A Scalable Simulation Platform with Auto-Generated Embodied Supermarket Environments — Hu et al. (2025) (arXiv:2511.21161, 2025)

What this evaluates

Evaluates embodied agents and multimodal LLMs on long-horizon manipulation tasks in procedurally generated supermarket environments. Specifically, it probes spatial reasoning, occlusion handling, and collision avoidance during checkout unloading and in-aisle item collection.

Datasets

• MarketGen Benchmark — total ?; splits: test (-1)

Metrics

• Success Rate (SR) (primary) — range: percent
  • Percentage of trials where the agent successfully completes the full task objective (e.g., unloading all basket items or collecting the target shelf item) without violating constraints.
• Success Path Length (SPL) — range: percent
  • Composite efficiency metric combining task success with path optimality, reported as a percentage.
• Path Length (PL) — range: other
  • Total distance traveled by the agent during the trial, measured in meters.

Input / output format

Input: Visual observations (rendered images/point clouds) of the supermarket scene and natural language task instructions.

Output: Sequential robot control commands (navigation waypoints, grasp poses, placement actions) executed in the simulation or real robot.

Scoring recipe

def compute_sr(predictions, gold_tasks):
    successful = 0
    for pred, task in zip(predictions, gold_tasks):
        if pred.task_completed and pred.no_collision and pred.all_items_collected:
            successful += 1
    return (successful / len(predictions)) * 100

Common pitfalls

• Success Rate (SR) values are consistently low (<15%) for MLLMs, indicating that decoupled planning and affordance models struggle with high-clutter, long-horizon tasks.
• SPL and Path Length (PL) metrics are only reported for the Checkout Unloading track, preventing direct efficiency comparisons with the In-Aisle track.
• Sim-to-real validation uses a fixed set of 4 object shapes with only 20 trials each, which does not represent the full 1,000+ product asset library.

Evidence (verbatim from paper)

We evaluate the performance of several representative Multi-modal Large Language Models (MLLMs) on our two benchmark tracks: Checkout Unloading and In-Aisle Item Collection. As shown in Tab.[2], the results highlight the significant challenge posed by long-horizon, high-clutter tasks in realistic supermarket scenarios. While Gemini-2.5-Pro achieves the highest overall Success Rate (SR), its performance remains below 15%, indicating that the modular methods struggle with the complexity of these tasks.

Citation

@misc{hu2025marketgen,
  title={MarketGen: A Scalable Simulation Platform with Auto-Generated Embodied Supermarket Environments},
  author={Hu et al. (2025)},
  year={2025},
  note={arXiv:2511.21161}
}

• arXiv: 2511.21161