Flava Multimodal Vision Nlp Eval

flava-multimodal-vision-nlp-eval

FLAVA: A Foundational Language And Vision Alignment Model — Singh et al. (arXiv:2112.04482, 2021)

What this evaluates

Evaluates a unified vision-language foundation model across 35 downstream tasks spanning vision classification, natural language understanding, and multimodal reasoning/retrieval. It probes the model's ability to generalize from joint unimodal and multimodal pretraining to zero-shot and fine-tuned downstream settings.

Datasets

• GLUE (MNLI, CoLA, MRPC, QQP, SST-2, QNLI, RTE, STS-B) — total ?; splits: dev (-1)
• 22 Vision Datasets (ImageNet, Food101, CIFAR10, CIFAR100, Cars, Aircraft, DTD, Pets, Caltech101, Flowers102, MNIST, STL10, EuroSAT, GTSRB, KITTI, PCAM, UCF101, CLEVR, FER 2013, SUN397, SST, Country211) — total ?; splits: test (-1)
• VQAv2 — total ?; splits: test-dev (-1)
• SNLI-VE — total ?; splits: test (-1)
• Hateful Memes — total ?; splits: test (-1)
• Flickr30K — total ?; splits: test (-1)
• COCO — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correct predictions: sum(pred == gold) / total_samples. Used for most classification tasks across all modalities.
• macro_average — range: [0, 1]
  • Mean of the average scores across the three modality groups (vision, NLP, multimodal).
• F1 — range: [0, 1]
  • Harmonic mean of precision and recall. Reported alongside accuracy for MRPC and QQP.
• PCC — range: [-1, 1]
  • Pearson Correlation Coefficient. Used for CoLA.
• MCC — range: [-1, 1]
  • Matthews Correlation Coefficient. Used for STS-B.
• AUROC — range: [0, 1]
  • Area Under the Receiver Operating Characteristic curve. Used for Hateful Memes.
• Recall@1 — range: [0, 1]
  • Proportion of queries where the correct image/text is ranked first based on contrastive matching scores.
• Recall@5 — range: [0, 1]
  • Proportion of queries where the correct image/text is ranked in the top 5 based on contrastive matching scores.

Input / output format

Input: Single images for vision tasks, single text sequences for NLP tasks, and image-text pairs for multimodal tasks.

Output: Class labels, regression scores, or ranked lists of images/texts depending on the task. For retrieval tasks, outputs are matching scores used to rank candidates.

Scoring recipe

def compute_metrics(predictions, golds, task_type, metric_name):
    if task_type == 'classification':
        if metric_name == 'accuracy':
            return sum(p == g for p, g in zip(predictions, golds)) / len(golds)
        elif metric_name == 'F1':
            return f1_score(golds, predictions, average='macro')
        elif metric_name == 'AUROC':
            return roc_auc_score(golds, predictions)
    elif task_type == 'retrieval':
        k = int(metric_name.split('@')[1])
        return sum(g in p[:k] for p, g in zip(predictions, golds)) / len(golds)
    elif task_type == 'macro':
        return mean(avg_vision, avg_nlp, avg_multimodal)

Common pitfalls

• The macro average is computed across the three modality averages (vision, NLP, multimodal), not across all 35 individual tasks.
• GLUE results are reported on development sets, and specific metrics vary by task (e.g., PCC for CoLA, MCC for STS-B, F1/accuracy for MRPC/QQP).
• Retrieval tasks (Flickr30K, COCO) are evaluated zero-shot using contrastive matching scores, not fine-tuned.

Evidence (verbatim from paper)

We report the average performance on the NLP, vision, and multimodal tasks, and an additional macro average across all the three modalities in Table 3, and also the detailed the performance on each task in Table 4. We report PCC for CoLA, MCC for STS-B, and AUROC for Hateful Memes, respectively. We perform zero-shot text retrieval and image retrieval (TR and IR) on Flickr30K and COCO based on their matching scores from the contrastive loss and report top-1 and top-5 recall. For all other tasks we report accuracy.

Citation

@misc{singh2021flava,
  title={FLAVA: A Foundational Language And Vision Alignment Model},
  author={Singh et al.},
  year={2021},
  note={arXiv:2112.04482}
}

• arXiv: 2112.04482