Gridtopix Eval

gridtopix-eval

GridToPix: Training Embodied Agents with Minimal Supervision — Jain et al. (2021) (arXiv:2105.00931, 2021)

What this evaluates

Evaluates the ability of embodied agents to learn long-horizon planning and navigation tasks using only terminal rewards, and tests the effectiveness of distilling policies from simplified gridworld experts into visual agents via imitation learning.

Datasets

• PointGoal Navigation — total ?; splits: test (-1)
• Furniture Moving — total ?; splits: test (-1)
• 3 vs. 1 Football — total ?; splits: test (-1)

Metrics

• SPL (primary) — range: [0, 1]
  • SPL = (1/N) Σ (success_i × shortest_path_i / actual_path_i). Success is binary (1 if agent reaches target within threshold, else 0). MD-SPL averages path length only for successful trajectories.
• Success — range: [0, 1]
  • Binary metric indicating whether the agent reaches the target goal within a specified distance threshold.
• MD-SPL — range: [0, 1]
  • Mean Distance to Success Path Length. Computes SPL only over trajectories that successfully reach the goal, ignoring failures.
• Game Score — range: other
  • Task-specific scoring metric for the football environment, aggregating points from goals, assists, or other game events.

Input / output format

Input: Raw RGB pixels for visual agents; semantic top-down tensors or 1D state vectors for gridworld experts.

Output: Discrete or continuous action commands (e.g., move, turn, kick) to navigate or interact within the environment.

Scoring recipe

def compute_spl(predictions, gold):
    spl_scores = []
    for pred, gold_path in zip(predictions, gold):
        success = 1.0 if pred.reaches_target(gold_path) else 0.0
        shortest = gold_path.length()
        actual = pred.length()
        spl_scores.append(success * (shortest / actual) if actual > 0 else 0.0)
    return sum(spl_scores) / len(spl_scores)

Common pitfalls

• Comparing gridworld expert performance directly to visual agents, as the text explicitly warns they operate on different state spaces and serve only as loose upper bounds.
• Assuming direct reinforcement learning from terminal rewards works for visual agents without the proposed distillation step, as the paper shows DirectPix fails to learn meaningful policies under terminal rewards.
• Mixing up shaped and terminal reward results, as performance gaps and learning curves differ significantly between the two settings.

Evidence (verbatim from paper)

We report standard evaluation metrics on three tasks. To study sample efficiency, we also show learning curves. Terminal rewards (see Tab. 1 and Fig. 4). With perfect perception, gridworld experts can train to a high performance (e.g., 94% success in PointNav) and guide the learning of visual agents. In sharp contrast, DirectPix doesn't learn a meaningful policy in any of the tasks[11] – demonstrating present day methods' inability to learn from terminal rewards in these visual worlds. Our GRIDTOPIX variants perform significantly better. For instance, at PointNav, GRIDTOPIX obtains a respectable SPL of 0.638, inching towards the 0.788 obtained by the gridworld expert.

Citation

@misc{jain2021gridtopix,
  title={GridToPix: Training Embodied Agents with Minimal Supervision},
  author={Jain et al. (2021)},
  year={2021},
  note={arXiv:2105.00931}
}

• arXiv: 2105.00931