Memelens Multitask Eval

memelens-multitask-eval

MemeLens: Multilingual Multitask VLMs for Memes — Shahroor et al. (2026) (arXiv:2601.12539, 2026)

What this evaluates

Evaluates multimodal vision-language models on understanding memes across multiple languages and semantic categories. It probes cross-modal reasoning, cross-lingual transfer, and the ability to generalize across diverse tasks like harm detection, misinformation, and humor/sarcasm.

Datasets

• MemeLens Unified Benchmark — total ?; splits: test (-1)

Metrics

• Accuracy — range: [0, 1]
  • Proportion of correctly predicted instances out of the total number of instances.
• Macro-F1 (primary) — range: [0, 1]
  • Unweighted mean of the F1 scores for each class. Calculated as the average of recall and precision per class, then averaged across all classes to handle class imbalance.
• Weighted-F1 — range: [0, 1]
  • Mean of the F1 scores for each class, weighted by the number of true instances for each class (support).

Input / output format

Input: Multimodal meme instance consisting of an image and its embedded text.

Output: Discrete class label from the shared 20-task taxonomy. Explanation-augmented training also requires a natural language rationale.

Scoring recipe

def compute_metrics(y_true, y_pred):
    accuracy = (y_true == y_pred).mean()
    macro_f1 = f1_score(y_true, y_pred, average='macro')
    weighted_f1 = f1_score(y_true, y_pred, average='weighted')
    return accuracy, macro_f1, weighted_f1

Common pitfalls

• Source datasets use heterogeneous evaluation protocols and label granularities, requiring careful metric alignment (Macro-F1 vs Weighted-F1 vs POS-F1) for fair comparison.
• The unified benchmark filters out samples without embedded text, which alters the data distribution and reduces direct comparability with prior work on unfiltered meme datasets.
• Single-dataset fine-tuning causes over-specialization to specific annotation conventions, so performance on one benchmark does not indicate cross-task generalization.

Evidence (verbatim from paper)

We report Accuracy, Macro-F1 (the primary metric due to class imbalance), and Weighted-F1 for this evaluation and analysis.

Citation

@misc{shahroor2026memelens,
  title={MemeLens: Multilingual Multitask VLMs for Memes},
  author={Shahroor et al. (2026)},
  year={2026},
  note={arXiv:2601.12539}
}

• arXiv: 2601.12539