Distortbench Eval

distortbench-eval

DistortBench: Benchmarking Vision Language Models on Image Distortion Identification — Goyal et al. (2026) (arXiv:2604.19966, 2026)

What this evaluates

Evaluates vision-language models' ability to perform fine-grained low-level visual perception by identifying both the specific type of image distortion and its severity level from a single image. It probes whether models rely on direct perceptual pattern matching or struggle with subtle severity discrimination.

Datasets

• DistortBench — total 13500; splits: test (13500)

Metrics

• Acc. (primary) — range: percent
  • Answered-only joint accuracy: the percentage of correctly identified distortion type and severity level among all parseable responses. Calculated as (number of correct joint predictions / number of answered prompts) * 100.

Input / output format

Input: A single distorted image accompanied by a prompt requesting identification of the distortion type and its severity level.

Output: A JSON object containing the predicted distortion type and severity level.

Scoring recipe

def compute_acc(predictions, gold):
    answered = [(p, g) for p, g in zip(predictions, gold) if p is not None]
    if not answered:
        return 0.0
    correct = sum(1 for p, g in answered if p['type'] == g['type'] and p['level'] == g['level'])
    return (correct / len(answered)) * 100

Common pitfalls

• Thinking/CoT variants frequently produce unparseable responses (up to 3.4%) when they exhaust the token budget on reasoning steps; these are excluded from accuracy calculations, artificially inflating performance on a slightly easier effective subset.
• Total parameter count is a misleading metric for MoE models on this task; performance correlates better with active parameters and visual encoder quality rather than raw scale.
• Severity discrimination is the dominant failure mode for both models and humans, often causing same-type/wrong-level confusions even when distortion type is correctly identified.

Evidence (verbatim from paper)

Table 3: Main results. Answered-only joint accuracy (Acc.) with bootstrap 95% confidence intervals, type-balanced accuracy (T-Bal.), level-balanced accuracy (L-Bal.), and unparseable-response rate (Unp.) over the full 13,500-prompt evaluation.

Citation

@misc{goyal2026distortbench,
  title={DistortBench: Benchmarking Vision Language Models on Image Distortion Identification},
  author={Goyal et al. (2026)},
  year={2026},
  note={arXiv:2604.19966}
}

• arXiv: 2604.19966