Talk2car Eval

talk2car-eval

Talk2Car: Taking Control of Your Self-Driving Car — Deruyttere et al. (2019) (arXiv:1909.10838, 2019)

What this evaluates

Evaluates a model's ability to ground free-form natural language commands to specific objects in autonomous driving scenes. It probes spatial and relational language understanding, disambiguation of same-category objects, and handling of long-range referents and complex sentences.

Datasets

• Talk2Car — total 11959; splits: train (8349), val (1163), test (2447)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correctly identified referred objects. (Exact formula not specified in the provided dataset section; standard for object referral tasks.)

Input / output format

Input: A video frame (or short sequence) from an autonomous vehicle's front-facing camera, accompanied by a free-form natural language command referring to a specific object in the scene.

Output: Identification of the referred object, typically as a bounding box or object class/ID from the nuScenes annotation schema.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = 0
    for pred, gold_obj in zip(predictions, gold):
        if pred == gold_obj:
            correct += 1
    return correct / len(gold)
# Note: Exact scoring protocol not detailed in the provided text.

Common pitfalls

• Samples from the same video are strictly kept within a single split to prevent temporal data leakage.
• The test set is not a single monolithic set; it includes curated subsets for specific conditions (e.g., object distance, command length, category occurrence counts) to evaluate targeted capabilities.
• nuScenes test set 3D bounding boxes are undisclosed, so the Talk2Car test set relies on nuScenes training set annotations, which may affect generalization evaluation.

Evidence (verbatim from paper)

The Talk2Car dataset evaluates state-of-the-art models on disambiguating objects via spatial and relational language cues, handling complex sentences, distant objects, and real-time prediction speed—key challenges in real-world AV interactions. We have also identified multiple subsets of the test set, which allow evaluation of specific situations.

Citation

@misc{deruyttere2019talk2car,
  title={Talk2Car: Taking Control of Your Self-Driving Car},
  author={Deruyttere et al. (2019)},
  year={2019},
  note={arXiv:1909.10838}
}

• arXiv: 1909.10838