Touchdown Eval

touchdown-eval

Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environments — Chen et al. (2018) (arXiv:1811.12354, 2018)

What this evaluates

Evaluates an agent's capacity to navigate through street-view environments using natural language instructions and resolve complex spatial descriptions to locate a hidden target object within a panoramic image.

Datasets

• Touchdown — total 9326; splits: dev (-1), test (-1); repo https://github.com/lil-lab/touchdown

Metrics

• pixel distance (primary) — range: pixels
  • Computed as the Euclidean distance in pixels between the predicted target location and the ground truth location. Accuracy is also reported as the fraction of predictions within an 80-pixel threshold.

Input / output format

Input: A 3×84×84 RGB perspective image (projected from a 360° panorama centered on the agent's heading) paired with a natural language instruction describing navigation and spatial relations.

Output: A probability distribution over the image pixels, with the highest probability pixel indicating the predicted target location.

Scoring recipe

def score(predictions, golds):
    distances = [np.linalg.norm(p - g) for p, g in zip(predictions, golds)]
    acc = sum(1 for d in distances if d <= 80) / len(distances)
    return {'accuracy': acc, 'mean_pixel_distance': np.mean(distances)}

Common pitfalls

• Models frequently latch onto visually salient or high-frequency objects (e.g., red signs, trash cans) rather than resolving precise spatial relations.
• Allocentric spatial reasoning is required, but models often fail when the scene contains dynamic changes or when the reference object is not uniquely identifiable by name alone.

Evidence (verbatim from paper)

The selected pixel is 104px from the correct one. ... the highest probability pixel is close enough (i.e., within 80 pixels) of the correct location.

Citation

@misc{chen2018touchdown,
  title={Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environments},
  author={Chen et al. (2018)},
  year={2018},
  note={arXiv:1811.12354}
}

• arXiv: 1811.12354