qhjqhj00 mewl-eval
MEWL: Few-shot multimodal word learning with referential uncertainty — Jiang et al. (2023) (arXiv:2306.00503, 2023)
What this evaluates
This benchmark probes few-shot multimodal word learning under referential uncertainty by testing cross-situational reasoning, semantic bootstrapping, and pragmatic inference. It evaluates how well vision-language and language models generalize from limited examples to name attributes, objects, relations, numbers, and pragmatic concepts compared to human baselines.
Datasets
- MEWL — total ?; splits: train (-1), test (-1); repo https://github.com/jianggy/MEWL
Metrics
accuracy(primary) — range: percent- Standard classification accuracy calculated as the proportion of correct predictions out of total test instances. Reported as a percentage per task and averaged across all nine tasks.
Input / output format
Input: Multimodal input consisting of one or more reference images and a text prompt/caption. The text prompt varies by task type: object-centric captions for attribute naming, relative spatial descriptions for relational tasks, and ground-truth pointing cues for pragmatic tasks.
Output: A single word or class label corresponding to the target concept (e.g., shape, color, material, object, composite, relation, bootstrap, number, or pragmatic term).
Scoring recipe
```python
def compute_accuracy(predictions, gold_labels):
correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
return (correct / len(gold_labels)) * 100
# Per-task accuracy
per_task_scores = {task: compute_accuracy(preds[task], golds[task]) for task in tasks}
# Overall average
overall_avg = sum(per_task_scores.values()) / len(per_task_scores)
## Common pitfalls
- The benchmark simplifies the word learning challenge by injecting strong human bias via detailed captions and ground-truth pointing, which circumvents true concept abstraction and makes it more translation-like.
- GPT-3.5 is evaluated zero-shot via API (temperature 0), while all other models are fine-tuned on the training sets, creating an unfair baseline comparison.
- Human participants are evaluated on a randomly drawn 10-question subset from the test set per task, whereas models are evaluated on the full test set, complicating direct score comparison.
## Evidence (verbatim from paper)
> We report the model performance on the test sets. Table 2 summarizes the performance of both machines and humans, with result visualization in Figure 3. For each of the nine tasks, every participant was presented with a randomly drawn ten-question subset from the task's test set.
## Citation
```bibtex
@misc{jiang2023mewl,
title={MEWL: Few-shot multimodal word learning with referential uncertainty},
author={Jiang et al. (2023)},
year={2023},
note={arXiv:2306.00503}
}
- arXiv: 2306.00503