Coda Lm Eval

coda-lm-eval

Automated Evaluation of Large Vision-Language Models on Self-driving Corner Cases — Chen et al. (2024) (arXiv:2404.10595, 2024)

What this evaluates

Evaluates Large Vision-Language Models on real-world autonomous driving corner cases across three tasks: general perception, regional perception, and driving suggestions. It probes the model's ability to accurately identify traffic-relevant objects, explain their impact on driving behavior, and generate actionable, rational driving advice in complex scenarios.

Datasets

• CODA-LM — total ?; splits: test (-1)

Metrics

• Text-Score (primary) — range: [1, 100]
  • A 1-10 scale rating generated by a text-only GPT-4 judge, multiplied by 10 to normalize to a 1-100 range. The judge evaluates accuracy, hallucination suppression, and correlation of reasons based on task-specific prompts.
• BLEU-4 — range: [1, 100]
  • Lexical n-gram overlap metric. Scores are multiplied by 100 to normalize to a 1-100 range. Noted as failing to capture semantic accuracy.
• METEOR — range: [1, 100]
  • Metric accounting for synonyms and stemming. Scores are multiplied by 100 to normalize to a 1-100 range. Noted as not reflecting actual semantics accurately.
• CIDEr — range: [1, 100]
  • Consensus-based Image Description Evaluation metric. Scores are multiplied by 100 to normalize to a 1-100 range. Noted as unsuitable for texts with low lexical repetition.
• SPICE — range: [1, 100]
  • Semantic Propositional Image Caption Evaluation metric. Scores are multiplied by 100 to normalize to a 1-100 range. Noted as reflecting semantic accuracy to some extent.

Input / output format

Input: An image/frame from a driving scenario paired with a task-specific prompt (general perception, regional perception, or driving suggestions). For automated judging, the input to the judge includes a system prompt, few-shot examples, and the model's generated response.

Output: A text description of objects, their impact on driving, or driving suggestions. The automated judge must output a rating strictly in the format: Rating: [[X]] (where X is an integer from 1 to 10).

Scoring recipe

def compute_text_score(predictions, references, few_shot_samples):
    scores = []
    for pred, ref in zip(predictions, references):
        query = f"{SYSTEM_PROMPT}\nFew-shot: {few_shot_samples}\nRef: {ref}\nPred: {pred}"
        response = gpt4_judge(query)
        rating = int(re.search(r'\[\[(\d+)\]\]', response).group(1))
        scores.append(rating * 10)
    return sum(scores) / len(scores)

Common pitfalls

• Traditional lexical metrics (BLEU-4, CIDEr) perform poorly on driving descriptions due to low lexical overlap, misleading performance assessment.
• Judge bias toward response length can inflate scores; prompts explicitly instruct to ignore length.
• Hallucination of non-existent objects is a common failure mode requiring explicit suppression checks in the evaluation criteria.

Evidence (verbatim from paper)

When conducting a corner case regional perception evaluation, the data is organized in the form of brief sentences. Therefore, in addition to using the Text-Score for evaluation, we also explore the impact of traditional keyword-based metrics, including BLEU-4 [40], METEOR [5], CIDEr [46], and SPICE [2], as shown in Tab. 7. For better demonstration, we multiplie the scores by 100, normalizing them to a range of 1-100, similarly with the Text-Score. BLEU-4 primarily evaluates quality through lexical matching and cannot capture the semantic accuracy of the generated text. CIDEr is not suitable for texts with low lexical repetition. Hence, the scores from these two metrics do not reflect performance accurately. Although METEOR can account for synonyms, it still does not reflect the actual semantics, so despite some differences in scores, they are not accurate. In contrast, SPICE can reflect semantic accuracy to some extent, and even though the overall scores are still low, it successfully indicates the trend among different models, with InternLM2-v1 still leading among open-source models. By default, we still adopt the Text-Score as the primary evaluation metric, unless otherwise spec

Citation

@misc{chen2024coda,
  title={Automated Evaluation of Large Vision-Language Models on Self-driving Corner Cases},
  author={Chen et al. (2024)},
  year={2024},
  note={arXiv:2404.10595}
}

• arXiv: 2404.10595