Tib Bench Eval

tib-bench-eval

Summarization of Multimodal Presentations with Vision-Language Models: Study of the Effect of Modalities and Structure — Gigant et al. (2025) (arXiv:2504.10049, 2025)

What this evaluates

Evaluates vision-language models on their ability to generate accurate and linguistically coherent summaries of text-heavy multimodal presentations. It probes cross-modal alignment, long-context understanding, and the impact of different input modalities (raw video, slides, transcripts, interleaved pairs) and token budgets on summarization quality.

Datasets

• TIB-bench — total ?; splits: test (-1)

Metrics

• $R_1$ — range: percent
  • ROUGE-1 computes unigram F1 overlap between generated and reference summaries. Reported as a percentage.
• $R_2$ — range: percent
  • ROUGE-2 computes bigram F1 overlap between generated and reference summaries. Reported as a percentage.
• $G$ — range: percent
  • GRUEN evaluates linguistic quality using a learned scoring model. Reported on a 0–100 scale.
• $IbR_{transcript}$ — range: percent
  • Importance-based relevance score measuring summary relevance weighted by transcript content importance. Exact computation defined in Section 3.4.
• $IbR_{ocr}$ — range: percent
  • Importance-based relevance score measuring summary relevance weighted by OCR-extracted text importance. Exact computation defined in Section 3.4.
• $IbR_{overall}$ (primary) — range: percent
  • Overall importance-based relevance score measuring summary relevance weighted by combined source content importance. Exact computation defined in Section 3.4.

Input / output format

Input: Multimodal presentation data provided as raw video, extracted slides, transcripts, or interleaved slide-transcript pairs. Inputs are tokenized with a specified visual token budget (e.g., 64–512 tokens) and truncated to 16k tokens if necessary to prevent out-of-memory errors.

Output: Natural language summary of the multimodal presentation.

Scoring recipe

def compute_metrics(predictions, references):
    # Metrics computed as defined in Section 3.4
    r1 = rouge_score(predictions, references, rouge_types=['rouge1']) * 100
    r2 = rouge_score(predictions, references, rouge_types=['rouge2']) * 100
    g = gruen_score(predictions, references) * 100
    ibr_trans = ibr_score(predictions, references, modality='transcript')
    ibr_ocr = ibr_score(predictions, references, modality='ocr')
    ibr_overall = ibr_score(predictions, references, modality='overall')
    return {
        'R1': r1, 'R2': r2, 'G': g,
        'IbR_transcript': ibr_trans, 'IbR_ocr': ibr_ocr, 'IbR_overall': ibr_overall
    }

Common pitfalls

• Models may fail to generate coherent summaries if inputs exceed their native training context length (e.g., Idefics-2 struggles beyond 2k tokens), making context length a critical confounding factor.
• 4-bit quantization and input truncation to 16k tokens are applied to prevent OOM errors but can significantly degrade performance, especially for models not explicitly trained on long contexts.
• IbR metrics require modality-specific importance weighting; standard ROUGE/GRUE scores alone do not capture cross-modal alignment or content relevance.

Evidence (verbatim from paper)

We evaluate the summaries generated by these models with the collection of metrics described in Section [3.4]. The results are reported in Table [1]. ... The importance-based relevance ($IbR$) scores reveal that with these models, visual and textual understanding are improved jointly.

Citation

@misc{gigant2025summarizationmultimodal,
  title={Summarization of Multimodal Presentations with Vision-Language Models: Study of the Effect of Modalities and Structure},
  author={Gigant et al. (2025)},
  year={2025},
  note={arXiv:2504.10049}
}

• arXiv: 2504.10049