Groundlie360 Eval

groundlie360-eval

A New Dataset and Benchmark for Grounding Multimodal Misinformation — Bingjian Yang et al. (2025) (arXiv:2509.08008, 2025)

What this evaluates

This benchmark evaluates a model's ability to detect and localize multimodal misinformation across text, speech, and video. It probes fine-grained cross-modal reasoning by requiring binary veracity classification, sub-type categorization, and precise grounding of fake content at the token, frame, and bounding-box levels.

Datasets

• GroundLie360 — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/yangbingjian/GroundLie360

Metrics

• F1 score (primary) — range: [0, 1]
  • Harmonic mean of Precision and Recall: 2 * (Prec * Rec) / (Prec + Rec). Computed per task (binary, subtype, token, frame) and averaged macro/micro as specified.
• Precision, Recall — range: [0, 1]
  • Precision: fraction of predicted positives that are correct. Recall: fraction of actual positives correctly identified. Reported for classification and grounding subtasks.
• m_tIoU, m_vIoU, vIoU@0.3, vIoU@0.5 — range: [0, 1]
  • Mean temporal IoU and mean visual IoU measure overlap between predicted and ground-truth temporal/spatial regions. vIoU@threshold reports the proportion of predictions where IoU exceeds the threshold.

Input / output format

Input: Multimodal news instances containing text, audio/speech, and video. Videos are scene-segmented and uniformly sampled to 16 frames per scene. Text inputs are limited to 1024 tokens. The model receives a question-driven prompt to perform classification and grounding.

Output: Per instance: (1) binary veracity label (real/fake), (2) misinformation sub-type label (e.g., false title, temporal edit, CGI, false speech, contradictory, unsupported), (3) token-level text spans, (4) frame indices for temporal grounding, (5) bounding boxes for spatial grounding.

Scoring recipe

def compute_classification_metrics(preds, golds):
    tp = sum(1 for p, g in zip(preds, golds) if p == g)
    fp = sum(1 for p, g in zip(preds, golds) if p != g)
    fn = sum(1 for p, g in zip(preds, golds) if p != g)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    return f1, prec, rec

def compute_grounding_iou(pred_regions, gold_regions):
    ious = []
    for pred, gold in zip(pred_regions, gold_regions):
        inter = intersection_area(pred, gold)
        union = union_area(pred, gold)
        ious.append(inter / union if union > 0 else 0)
    return sum(ious) / len(ious) if ious else 0

Common pitfalls

• Error propagation: Mistakes in upstream binary or sub-type classification stages directly magnify failures in downstream localization.
• Sample imbalance and ambiguity: Uneven distribution across fake types and fragmented spoken language cause significant performance drops for categories like false speech and temporal edits.
• Multi-stage pipeline dependency: Grounding accuracy is tightly coupled with classification confidence, making isolated grounding evaluation misleading without considering upstream errors.

Evidence (verbatim from paper)

We use different evaluation metrics depending on the task type: (1) Binary veracity classification: Precision, Recall, and F1 score. (2) Sub-type classification: Macro-averaged Precision, Recall, and F1 score. (3) Grounding tasks:(a) Textual grounding: Token-level Precision, Recall, and F1 score; (b) Video temporal grounding: Frame-level Precision, Recall, and F1 score; (c) Video spatial-temporal grounding: Mean temporal IoU (m_tIoU), mean visual IoU (m_vIoU), and vIoU@0.3 / vIoU@0.5

Citation

@misc{yang2025groundlie360,
  title={A New Dataset and Benchmark for Grounding Multimodal Misinformation},
  author={Bingjian Yang et al. (2025)},
  year={2025},
  note={arXiv:2509.08008}
}

• arXiv: 2509.08008