Nim Benchmark Eval

nim-benchmark-eval

Finding Needles in Images: Can Multimodal LLMs Locate Fine Details? — Parth Thakkar et al. (arXiv:2508.05053, 2025)

What this evaluates

Evaluates multimodal LLMs' ability to locate and reason about fine-grained details in complex real-world documents. It specifically probes resilience against irrelevant information (distractor images) and measures performance across open- and closed-domain retrieval settings.

Datasets

• ArxiVQA — total ?; splits: test (500)
• DUDE — total ?; splits: test (500)
• NiM-Benchmark — total 937; splits: test (937)

Metrics

• Exact-Match (EM) (primary) — range: [0, 1]
  • 1 if the predicted answer string exactly matches the gold answer string, 0 otherwise.
• F1-Score — range: [0, 1]
  • Token-level F1 score computed as the harmonic mean of precision and recall between predicted and gold answers.
• ANLS Score — range: [0, 1]
  • Average Normalized Levenshtein Similarity; computes character-level overlap normalized by the length of the gold answer.
• Accuracy — range: [0, 1]
  • Fraction of correctly answered multiple-choice questions, used specifically for ArxiVQA.

Input / output format

Input: Question text and one or more context images. In open-domain settings, top-k retrieved images are provided; in closed-domain, a predefined set containing the exact context. NiM-Benchmark additionally includes distractor images.

Output: Textual answer string generated by the model.

Scoring recipe

def compute_metrics(predictions, golds, dataset_name):
    em = [1.0 if p.strip() == g.strip() else 0.0 for p, g in zip(predictions, golds)]
    f1 = [token_f1(p, g) for p, g in zip(predictions, golds)]
    anls = [char_levenshtein_sim(p, g) for p, g in zip(predictions, golds)]
    if dataset_name == 'ArxiVQA':
        return {'accuracy': sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)}
    return {'EM': np.mean(em), 'F1': np.mean(f1), 'ANLS': np.mean(anls)}

Common pitfalls

• ArxiVQA evaluation uses the training set because the official test split is unavailable, which may skew results compared to standard benchmarks.
• Hyperparameters are tuned on a small random subset of 50 questions per dataset, potentially leading to overfitting to the tuning set.
• NiM-Benchmark includes distractor images to test resilience; many baseline models fail to filter irrelevant context, causing performance drops.

Evidence (verbatim from paper)

We use Exact-Match (EM), F1-Score (Rajpurkar, 2016), and ANLS Score (Biten et al., 2019) as automatic metrics to assess the correctness of the predicted answers. For ArxiVQA, being a multiple-choice question dataset, we use accuracy as the evaluation metric.

Citation

@misc{thakkar2025findingneedles,
  title={Finding Needles in Images: Can Multimodal LLMs Locate Fine Details?},
  author={Parth Thakkar et al.},
  year={2025},
  note={arXiv:2508.05053}
}

• arXiv: 2508.05053