Lingoqa Eval

lingoqa-eval

Efficient Visual Question Answering Pipeline for Autonomous Driving via Scene Region Compression — Cai et al. (2026) (arXiv:2601.07092, 2026)

What this evaluates

Evaluates vision-language models on autonomous driving video question answering, testing their ability to understand temporal visual context, describe scenes, predict actions, and justify answers based on driving scenarios.

Datasets

• LingoQA — total 419900; splits: train (-1), val (-1), test (-1)

Metrics

• Ling-Judge (primary) — range: [0, 100]
  • A semantic similarity score computed by a fine-tuned BERT-based classifier that compares the generated answer against the ground truth answer. Scores are reported on a 0-100 scale.
• BLEU — range: [0, 1]
  • Standard N-gram based metric measuring the precision of n-gram overlaps between the generated answer and ground truth, typically averaged across n=1 to 4.
• FLOPs (%) — range: percent
  • Relative computational cost measured as a percentage of the baseline full-frame (5-frame) inference cost.

Input / output format

Input: A sequence of 1 to 5 video frames from an autonomous driving scenario, paired with a natural language question.

Output: A natural language text answer generated by the model.

Scoring recipe

def compute_metrics(predictions, golds):
    bleu_scores = [sentence_bleu(g, p) for g, p in zip(golds, predictions)]
    ling_scores = []
    for g, p in zip(golds, predictions):
        score = bert_classifier.predict_semantic_similarity(g, p)
        ling_scores.append(score)
    return {'BLEU': mean(bleu_scores), 'Ling-Judge': mean(ling_scores)}

Common pitfalls

• Ling-Judge requires a specific fine-tuned BERT classifier; using a vanilla BERT or standard semantic similarity models will yield incorrect scores.
• BLEU scores are inherently low (10-15) for open-ended video QA; comparing absolute BLEU values across different datasets or tasks is misleading.
• Performance is highly sensitive to the number of input frames (1 vs 5); results must be compared within the same frame-count setting.

Evidence (verbatim from paper)

Besides the common N-Gram-based metrics such as BLEU [[11]], LingoQA introduces a semantic-based metric, $Lingo$-$Judge$, such that it adopt fine-tuned Bert-based model as classifier to evaluate the semantic similarity between the generated answer and the ground truth answer.

Citation

@misc{cai2026efficient,
  title={Efficient Visual Question Answering Pipeline for Autonomous Driving via Scene Region Compression},
  author={Cai et al. (2026)},
  year={2026},
  note={arXiv:2601.07092}
}

• arXiv: 2601.07092