Mint 1t Eval

mint-1t-eval

MINT-1T: Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokens — Awadalla et al. (2024) (arXiv:2406.11271, 2024)

What this evaluates

Evaluates the multimodal interleaved reasoning and in-context learning capabilities of large multimodal models (LMMs) across image captioning, visual question answering, and multi-image reasoning tasks.

Datasets

• COCO (Karpathy test) — total ?; splits: test (-1)
• TextCaps — total ?; splits: validation (-1)
• VQAv2 — total ?; splits: validation (-1)
• OK-VQA — total ?; splits: validation (-1)
• TextVQA — total ?; splits: validation (-1)
• VizWiz — total ?; splits: validation (-1)
• MMMU — total ?; splits: test (-1)
• Mantis-Eval — total ?; splits: test (-1)

Metrics

• scores (primary) — range: percent / CIDEr
  • Accuracy for VQA and reasoning benchmarks; CIDEr for image captioning benchmarks. Scores are averaged over multiple evaluation runs where demonstrations are randomized.

Input / output format

Input: One or more images interleaved with text, plus a prompt containing k-shot demonstrations (k=1, 4, or 8) sampled from the training set, and the target question/image.

Output: Text string (caption or answer).

Scoring recipe

def evaluate(model, dataset, shots=4, num_runs=3):
    all_scores = []
    for run in range(num_runs):
        demos = random_sample(dataset.train, shots)
        preds = [model.generate(demos + inst) for inst in dataset.test]
        if dataset.task == 'captioning':
            score = compute_cider(preds, dataset.test.gold)
        else:
            score = compute_accuracy(preds, dataset.test.gold)
        all_scores.append(score)
    return mean(all_scores), std(all_scores)

Common pitfalls

• Performance is highly sensitive to prompt choice; authors had to ablate prompts and select the best-performing one per task.
• Scores vary based on which training examples are used as demonstrations; averaging over multiple random seeds/runs is required for stable comparison.
• Different model architectures (e.g., XGen-MM vs Idefics2) show divergent trends on captioning vs VQA, so results are not architecture-agnostic.

Evidence (verbatim from paper)

Our reported scores are averaged over multiple evaluation runs where we randomize demonstrations. We find that performance is sensitive to the chosen prompts, so we ablate through different prompts for each task and choose the prompt that performs best.

Citation

@misc{awadalla2024mint1t,
  title={MINT-1T: Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokens},
  author={Awadalla et al. (2024)},
  year={2024},
  note={arXiv:2406.11271}
}

• arXiv: 2406.11271