Shredbench Eval

shredbench-eval

ShredBench: Evaluating the Semantic Reasoning Capabilities of Multimodal LLMs in Document Reconstruction — Guo et al. (2026) (arXiv:2604.23813, 2026)

What this evaluates

Evaluates multimodal LLMs' ability to reconstruct shredded documents from fragmented visual inputs. It probes cross-modal semantic reasoning, visual discontinuity alignment, and fine-grained positional continuity awareness across natural language, source code, and tabular data.

Datasets

• ShredBench — total ?; splits: test (-1); repo https://github.com/ythere-y/ShredBench

Metrics

• NED (primary) — range: [0, 1]
  • Normalized Edit Distance. Computed as the Levenshtein distance between prediction Ŷ and ground truth Y divided by the maximum length of the two strings: NED(Y,Ŷ) = Lev(Y,Ŷ) / max(|Y|,|Ŷ|). Lower values indicate higher similarity.
• BLEU — range: [0, 1]
  • Bilingual Evaluation Understudy. Calculates the geometric mean of n-gram precision, penalized for brevity using a Brevity Penalty (BP) based on generated and reference lengths.
• ROUGE-L — range: [0, 1]
  • Captures sentence-level structure via the Longest Common Subsequence (LCS). Computes the weighted F-measure of LCS precision and recall, where beta controls the precision-recall trade-off.
• TEDS — range: [0, 1]
  • Tree-Edit-Distance-based Similarity. Used for tables, it models content as trees (e.g., HTML DOM) and computes 1 - TED(T,Ť) / max(|T|,|Ť|). Higher scores indicate better structural and content reconstruction.

Input / output format

Input: Images of shredded document fragments (8, 12, or 16 pieces) accompanied by a prompt instructing the model to reconstruct the original document content.

Output: Reconstructed text, code, or table in plain text or Markdown/HTML format.

Scoring recipe

def compute_metrics(pred, ref, is_table=False):
    ned = levenshtein_distance(pred, ref) / max(len(pred), len(ref))
    bleu = compute_bleu(pred, ref)
    rouge_l = compute_rouge_l_fscore(pred, ref)
    if is_table:
        teds = 1 - tree_edit_distance(pred, ref) / max(len(pred), len(ref))
        return ned, bleu, rouge_l, teds
    return ned, bleu, rouge_l

Common pitfalls

• Chinese text scores are disproportionately penalized by BLEU and ROUGE due to the lack of explicit word delimiters, causing minor errors to disrupt segmentation boundaries.
• Python code reconstruction fails more frequently than Java or C++ because shredding disrupts whitespace-dependent indentation, whereas explicit delimiters in other languages act as visual anchors.
• Table reconstruction requires rigid 2D spatial alignment; models optimized for semantic flow may underperform on tabular data despite high semantic accuracy, as shown by divergences between NED and TEDS.

Evidence (verbatim from paper)

We employ Normalized Edit Distance (NED) for general text similarity. It normalizes the Levenshtein distance (Lev) between prediction Ŷ and ground truth Y: NED(Y,Ŷ) = Lev(Y,Ŷ) / max(|Y|,|Ŷ|). A lower NED implies higher similarity.

Citation

@misc{guo2026shredbench,
  title={ShredBench: Evaluating the Semantic Reasoning Capabilities of Multimodal LLMs in Document Reconstruction},
  author={Guo et al. (2026)},
  year={2026},
  note={arXiv:2604.23813}
}

• arXiv: 2604.23813