Re Verse Eval

re-verse-eval

Re:Verse -- Can Your VLM Read a Manga? — Baranwal et al. (2025) (arXiv:2508.08508, 2025)

What this evaluates

Evaluates vision-language models' ability to comprehend long-form sequential manga narratives, focusing on story synthesis, character grounding, and temporal reasoning across non-linear, multi-panel sequences.

Datasets

• Re:Verse — total 308; splits: test (308); repo https://github.com/eternal-f1ame/Re-Verse

Metrics

• BERTScore (primary) — range: [0, 1]
  • Computes cosine similarity between contextual embeddings of generated and reference text using bert-base-uncased, then aggregates precision, recall, and F1 scores.
• ROUGE — range: [0, 1]
  • Measures n-gram overlap between generated and reference texts, typically reporting ROUGE-1 and ROUGE-L recall and F1.
• STTR — range: [0, 1]
  • Sliding Text-to-Text Ratio; uses a sliding window (size 50, step 10) to assess text quality and fluency across sequences.
• Lexical density — range: [0, 1]
  • Ratio of content words to total word count in the generated narrative.
• NER density — range: [0, 1]
  • Ratio of named entity mentions to total word count, validated with spaCy and story-specific terminology.
• Accuracy — range: percent
  • Percentage of correctly predicted next/intermediate pages or correctly answered VQA questions.

Input / output format

Input: Manga pages (resized to 224× width, preserving aspect ratio) with spatial annotations (bounding boxes, semantic tags for text bubbles/thoughts) and corresponding light novel passages or sequential page contexts.

Output: For generation/summarization: coherent prose narratives or summaries. For grounding: spatial locations, semantic types, and character attributions. For temporal/VQA: predicted page indices or text answers.

Scoring recipe

def score(predictions, gold):
    scores = {}
    # Generation/Summarization
    scores['bertscore'] = compute_bertscore(predictions['text'], gold['text'], model='bert-base-uncased')
    scores['rouge'] = compute_rouge(predictions['text'], gold['text'])
    scores['sttr'] = compute_sttr(predictions['text'], window=50, step=10)
    scores['lexical_density'] = count_content_words(predictions['text']) / len(predictions['text'].split())
    scores['ner_density'] = count_ner_entities(predictions['text'], nlp=spacy_model) / len(predictions['text'].split())
    # Temporal/VQA
    scores['accuracy'] = sum(1 for p, g in zip(predictions['pred'], gold['gold']) if p == g) / len(gold['gold'])
    # Apply proportional penalties and thresholds as defined in protocol
    return apply_penalty_system(scores)

Common pitfalls

• Models often mistake surface-level visual-textual recognition for deep narrative comprehension, failing on long-range coherence and causal inference.
• Character attribution errors are frequent due to over-reliance on pronouns/honorifics without proper visual grounding or series-specific knowledge.
• Divergence between LLM judges (GPT-4o) and automated lexical metrics can obscure true narrative quality gaps.

Evidence (verbatim from paper)

We employ both an LLM (GPT-4o [[22]]) and automated metrics (BERTScore, ROUGE, STTR, lexical density, NER density), with statistical significance testing and embedding-based cross-modal alignment (details in supplementary). Temporal reasoning tasks use accuracy measures within VQA frameworks.

Citation

@misc{baranwal2025revers,
  title={Re:Verse -- Can Your VLM Read a Manga?},
  author={Baranwal et al. (2025)},
  year={2025},
  note={arXiv:2508.08508}
}

• arXiv: 2508.08508