Multimodal Oil Gas Framing Eval

multimodal-oil-gas-framing-eval

A Multimodal Benchmark for Framing of Oil & Gas Advertising and Potential Greenwashing Detection — Morio et al. (2025) (arXiv:2510.21679, 2025)

What this evaluates

This benchmark evaluates vision-language models' ability to perform multi-label framing classification on real-world oil and gas advertising videos. It probes multimodal understanding of implicit strategic communication, cultural context, and greenwashing detection across different video lengths and geographic regions.

Datasets

• Multimodal Oil & Gas Advertising Benchmark — total 706; splits: train (-1), test (-1); repo https://github.com/climate-nlp/multimodal-oil-gas-benchmark

Metrics

• F-score (primary) — range: percent
  • Micro-averaged F-score across all 13 framing labels for multi-label classification. Computed as the harmonic mean of micro-averaged precision and recall over the entire test set.

Input / output format

Input: A video represented by up to N_Frame sampled frame images (selected at mean timestamps of transcript segments) and corresponding transcript text, accompanied by annotation instructions. Optionally includes one in-context training example.

Output: A JSON list of predicted framing labels.

Scoring recipe

def compute_micro_fscore(predictions, gold):
    tp = fp = fn = 0
    for pred_set, gold_set in zip(predictions, gold):
        tp += len(pred_set & gold_set)
        fp += len(pred_set - gold_set)
        fn += len(gold_set - pred_set)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return f1 * 100  # reported as percent

Common pitfalls

• Frame sampling strategy (N_Frame=10 vs 3) and transcript inclusion significantly impact performance; omitting transcripts degrades results, especially on Facebook.
• Shorter videos (0-60s) are consistently harder to classify due to vague contextual cues, leading to lower F-scores regardless of model size.
• Models exhibit strong cultural/geographic biases; performance varies significantly based on the headquarters region of the advertising entity.

Evidence (verbatim from paper)

Our task is a multi-label classification setting, where the input is a video and the output is a set of predicted labels (and the label set varies depending on the domain). We use the F-score to evaluate classification performance.

Citation

@misc{morio2025multimodal,
  title={A Multimodal Benchmark for Framing of Oil & Gas Advertising and Potential Greenwashing Detection},
  author={Morio et al. (2025)},
  year={2025},
  note={arXiv:2510.21679}
}

• arXiv: 2510.21679