Mapdr Eval

mapdr-eval

Driving by the Rules: A Benchmark for Integrating Traffic Sign Regulations into Vectorized HD Map — Chang et al. (2024) (arXiv:2410.23780, 2024)

What this evaluates

Probes autonomous driving models' ability to extract lane-level traffic regulations from visual inputs and map them to vectorized HD map centerlines. It evaluates both rule extraction from image sequences and bipartite graph construction for rule-lane correspondence reasoning.

Datasets

• MapDR — total 18000; splits: unspecified (-1); repo https://github.com/MIV-XJTU/MapDR

Metrics

• correspondence status (primary) — range: [0, 1]
  • Accuracy of the predicted binary adjacency matrix E_pred against the ground truth E_gold, where each element indicates the correspondence status between a rule and a lane centerline.

Input / output format

Input: Image sequences X={x_i}{i=1}^n and local vectorized HD map centerlines L={l_i}{i=1}^k.

Output: For rule extraction: a set of rules R={r_i}_{i=1}^m in key:value pairs. For correspondence: a bipartite graph G=(R∪L,E) with a binary adjacency matrix E⊆{0,1}^{m×k}.

Scoring recipe

def score(predictions, gold):
    # predictions: list of rules, adjacency matrix E_pred (m x k)
    # gold: list of rules, adjacency matrix E_gold (m x k)
    # Align rule ordering if necessary, then compare adjacency matrices
    correct = (E_pred == E_gold).sum()
    total = E_pred.size
    return correct / total

Common pitfalls

• Confusing the two sub-tasks: rule extraction (visual-to-text) and correspondence reasoning (text-to-graph).
• Assuming a fixed number of rules (m) or lanes (k) per instance, as both vary dynamically.
• Treating the bipartite adjacency matrix as directed or weighted, whereas the protocol specifies a strictly binary correspondence status.

Evidence (verbatim from paper)

The final correspondence forms a bipartite graph G=(R∪L,E), where E⊆{0,1}^{m×k} and the element E_ij in the i-th row and j-th column of matrix E represents the corresponding status between r_i and l_j.

Citation

@misc{chang2024mapdr,
  title={Driving by the Rules: A Benchmark for Integrating Traffic Sign Regulations into Vectorized HD Map},
  author={Chang et al. (2024)},
  year={2024},
  note={arXiv:2410.23780}
}

• arXiv: 2410.23780