Provision Eval

provision-eval

ProVision: Programmatically Scaling Vision-centric Instruction Data for Multimodal Language Models — Zhang et al. (2024) (arXiv:2412.07012, 2024)

What this evaluates

Evaluates multimodal language models' vision-centric instruction following, multi-image reasoning, and general visual understanding. It probes the model's ability to process single and multiple images, answer factual questions, and perform complex reasoning across a suite of standard benchmarks.

Datasets

• CV-Bench (CVB-2D, CVB-3D) — total ?; splits: test (-1)
• SEED-Bench — total ?; splits: test (-1)
• MMBench (MMB) — total ?; splits: test (-1)
• MME — total ?; splits: test (-1)
• QBench2 — total ?; splits: test (-1)
• MMMU — total ?; splits: test (-1)
• RealWorldQA — total ?; splits: test (-1)
• MMStar — total ?; splits: test (-1)
• MMVet — total ?; splits: test (-1)
• Mantis-Eval — total ?; splits: test (-1)
• MMT-Bench (MMT) — total ?; splits: test (-1)
• TextVQA — total ?; splits: test (-1)

Metrics

• Avg. (primary) — range: percent
  • The arithmetic mean of the accuracy scores reported on each individual benchmark listed in the datasets array.

Input / output format

Input: One or more images paired with a text instruction or question.

Output: A short textual answer or a selected option from a multiple-choice list.

Scoring recipe

def compute_avg_score(predictions, golds, benchmarks):
    scores = {}
    for bench in benchmarks:
        correct = sum(1 for p, g in zip(predictions[bench], golds[bench]) if p == g)
        scores[bench] = correct / len(golds[bench]) * 100
    return sum(scores.values()) / len(scores)

Common pitfalls

• The choice between short-answer and multiple-choice formats significantly impacts performance; mixing them ('half-half') often yields the best results.
• Replacing base training data vs. augmenting it changes the optimization landscape, with augmentation generally providing more stable gains.
• Manual vs. model-generated scene graphs in the training data lead to different scaling behaviors, with manual graphs generally outperforming at smaller scales.

Evidence (verbatim from paper)

| CVB-2D | CVB-3D | SEED | MMB | MME | QBench2 | MMMU | RealWorldQA | Avg. |

Citation

@misc{zhang2024provision,
  title={ProVision: Programmatically Scaling Vision-centric Instruction Data for Multimodal Language Models},
  author={Zhang et al. (2024)},
  year={2024},
  note={arXiv:2412.07012}
}

• arXiv: 2412.07012