Zerosense Eval

zerosense-eval

ZeroSense:How Vision matters in Long Context Compression — Gao et al. (2026) (arXiv:2603.11846, 2026)

What this evaluates

Evaluates a model's ability to perform visual-text compression (OCR) by measuring raw text retention on rendered documents where the textual content has been deliberately stripped of semantic meaning. It isolates pure visual decoding capability from downstream linguistic priors or contextual inference.

Datasets

• ZeroSense — total ?; splits: test (-1); repo https://github.com/MedHK23/ZeroSense

Metrics

• text preservation capability (primary) — range: [0, 1]
  • Measures exact-match or character-level accuracy between the model's transcribed output and the ground-truth semantically irrelevant text. Designed to penalize models that rely on linguistic priors rather than visual decoding.

Input / output format

Input: Rendered document images containing layout-specific configurations (font size, bounding boxes, line height) populated with semantically agnostic text generated via low-probability token sampling.

Output: Transcribed text string corresponding to the content of the input image.

Scoring recipe

def score(predictions, gold):
    correct = 0
    total = len(gold)
    for pred, gt in zip(predictions, gold):
        if pred.strip() == gt.strip():
            correct += 1
    return correct / total if total > 0 else 0.0

Common pitfalls

• Models bypass visual decoding by using semantic priors to hallucinate or correct text when the input lacks meaningful context.
• Layout feature extraction from source datasets (Fox, Omni) may introduce rendering artifacts if bounding box reconstruction or font size estimation is inaccurate.
• The low-probability threshold τ for token sampling must be carefully tuned; too high retains semantics, too low may produce unrecognizable noise.

Evidence (verbatim from paper)

To prevent these semantic priors from biasing our measurement of the model’s text preservation capability, we propose the construction of an auxiliary dataset, termed the ZeroSense dataset. This auxiliary benchmark is designed to provide the precise rendering layouts $ heta$ (font size, case, capacity, line height, etc.) required by specific methods, while strictly guaranteeing the semantic irrelevance of the visual content. ... This creates a semantic vacuum that forces the evaluation to rely entirely on the visual retention of the rendered page rather than the language model’s prior knowledge.

Citation

@misc{gao2026zerosense,
  title={ZeroSense:How Vision matters in Long Context Compression},
  author={Gao et al. (2026)},
  year={2026},
  note={arXiv:2603.11846}
}

• arXiv: 2603.11846