Illusory Vqa Eval

illusory-vqa-eval

Illusory VQA: Benchmarking and Enhancing Multimodal Models on Visual Illusions — Rostamkhani et al. (2024) (arXiv:2412.08169, 2024)

What this evaluates

Evaluates multimodal models' ability to detect visual illusions (pareidolia) in images, comparing performance across raw, illusory, and low-pass filtered versions. It also measures zero-shot and fine-tuned OCR capabilities on text-containing illusion images to assess perceptual robustness and text recognition under distortion.

Datasets

• IllusionMNIST — total ?; splits: test (-1); repo https://github.com/IllusoryVQA/IllusoryVQA
• IllusionFashionMNIST — total ?; splits: test (-1); repo https://github.com/IllusoryVQA/IllusoryVQA
• IllusionAnimals — total ?; splits: test (-1); repo https://github.com/IllusoryVQA/IllusoryVQA
• IllusionChar — total ?; splits: test (-1); repo https://github.com/IllusoryVQA/IllusoryVQA

Metrics

• Accuracy (primary) — range: [0, 1]
  • Proportion of correctly classified instances: (TP + TN) / (TP + TN + FP + FN). Reported as a decimal or percentage.
• F1 — range: [0, 1]
  • Harmonic mean of Precision and Recall: 2 * (Precision * Recall) / (Precision + Recall).
• Precision — range: [0, 1]
  • Ratio of true positive predictions to all positive predictions: TP / (TP + FP).
• Recall — range: [0, 1]
  • Ratio of true positive predictions to all actual positives: TP / (TP + FN).
• WER — range: [0, 1]
  • Word Error Rate: minimum number of insertions, deletions, and substitutions of words to transform the predicted text into the ground truth, divided by the number of words in the ground truth.
• CER — range: [0, 1]
  • Character Error Rate: same as WER but computed at the character level.

Input / output format

Input: RGB images categorized as Raw (no illusion), Illusion (contains pareidolia), or Filtered (illusory images processed with Gaussian/blur low-pass filters). For IllusionChar, images containing text.

Output: Binary classification label (illusion vs. non-illusion) for IllusionMNIST, IllusionFashionMNIST, and IllusionAnimals; text string transcription for IllusionChar.

Scoring recipe

def compute_classification_metrics(preds, gold):
    tp = sum(1 for p, g in zip(preds, gold) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(preds, gold) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(preds, gold) if p == 0 and g == 1)
    tn = sum(1 for p, g in zip(preds, gold) if p == 0 and g == 0)
    acc = (tp + tn) / (tp + fp + fn + tn)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
    return {'accuracy': acc, 'precision': prec, 'recall': rec, 'f1': f1}

Common pitfalls

• Confusing 'Raw' images (non-illusion baselines) with 'Illusion' images when reporting or comparing detection performance.
• Human evaluation was only conducted on 'Illusion' images, making direct cross-condition comparison with models (evaluated on Raw/Illusion/Filtered) potentially unfair.
• API-based models (GPT-4o, Gemini) have varying coverage rates across datasets due to rate limits or content filters, which must be accounted for when interpreting accuracy scores.

Evidence (verbatim from paper)

Table 2: Zero-shot performance of different models on different datasets: The term ’Raw’ refers to raw images without any illusions. ’Illusion’ refers to illusory images, while ’Filtered’ indicates illusory images that have been processed with our filter. | | | Accuracy | Precision | Recall | F1 |

Citation

@misc{rostamkhani2024illusoryvqa,
  title={Illusory VQA: Benchmarking and Enhancing Multimodal Models on Visual Illusions},
  author={Rostamkhani et al. (2024)},
  year={2024},
  note={arXiv:2412.08169}
}

• arXiv: 2412.08169