Ervqa Eval

ervqa-eval

ERVQA: A Dataset to Benchmark the Readiness of Large Vision Language Models in Hospital Environments — Ray et al. (2024) (arXiv:2410.06420, 2024)

What this evaluates

This benchmark evaluates the clinical readiness of Large Vision Language Models (LVLMs) for emergency room monitoring tasks. It probes their ability to generate accurate, clinically cautious, and semantically entailed long-form answers from medical images, while identifying specific failure modes like hallucinations and overconfidence.

Datasets

• ERVQA — total ?; splits: test (-1); repo https://github.com/sourjyadip/ervqa-data

Metrics

• BLEU-1 — range: [0, 1]
  • Computes the unigram precision between the generated answer and the ground truth reference.
• ROUGE-L — range: [0, 1]
  • Measures the longest common subsequence between the generated answer and the ground truth reference.
• SentenceBERT Similarity — range: [0, 1]
  • Computes cosine similarity between sentence embeddings of the generated answer and the ground truth reference using a SentenceBERT model.
• Entailment Score (primary) — range: [0, 1]
  • ES = p(entailment | ref, gen), computed by applying a softmax function over the entailment class logit from a roberta-base-nli cross-encoder model, then averaged across all generations.
• CLIPScore Confidence — range: [0, 1]
  • CLIP-C = CLIP-S(img, gen) / (CLIP-S(img, ref) + CLIP-S(img, gen)), where CLIP-S computes cross-modal image-text similarity. Provides an image-grounded comparison of generated vs reference answers.

Input / output format

Input: A medical image (typically from an emergency room/hospital setting) paired with a long-form visual question.

Output: A generated long-form textual answer addressing the question.

Scoring recipe

def compute_metrics(predictions, references, images):
    bleu1 = bleu(predictions, references, n=1)
    rouge_l = rouge(predictions, references, l=True)
    sbert = sentencebert_similarity(predictions, references)
    es = mean([roberta_nli.softmax(entailment_logit(ref, gen)) for ref, gen in zip(references, predictions)])
    clip_c = mean([clip_score(img, gen) / (clip_score(img, ref) + clip_score(img, gen)) for img, ref, gen in zip(images, references, predictions)])
    return {'BLEU-1': bleu1, 'ROUGE-L': rouge_l, 'SentenceBERT': sbert, 'Entailment Score': es, 'CLIPScore Confidence': clip_c}

Common pitfalls

• Relying solely on lexical metrics (BLEU/ROUGE) misses clinical safety and semantic entailment, which are critical for healthcare applications.
• Error detection uses a finetuned BLIP-2 classifier trained on induced/silver-label data rather than human annotations, which may propagate model-specific biases.
• In-context learning requirements vary by model (e.g., Open-Flamingo needs few-shot even in 'zero-shot' settings), so strict prompt adherence is necessary for fair comparison.

Evidence (verbatim from paper)

To evaluate generated answers against ground truth, we use traditional model-free metrics such as BLEU-1, which compares unigrams, and ROUGE-L, which compares longest common subsequences. For semantic similarity, we use SentenceBERT Similarity... However, these metrics do not fully capture two essential factors for VQA evaluation: a) whether the ground truth answer is entailed within the generated answer, and b) whether the generated answer aligns with the image. To address these issues for our dataset, we adpapt two model based evaluation metrics suitable for our problem domain: Entailment Score: We use the roberta-base-nli model... We define Entailment score as follows: ES=p(entailment|ref,gen)

Citation

@misc{ray2024ervqa,
  title={ERVQA: A Dataset to Benchmark the Readiness of Large Vision Language Models in Hospital Environments},
  author={Ray et al. (2024)},
  year={2024},
  note={arXiv:2410.06420}
}

• arXiv: 2410.06420