Crisismmd Classification Eval

crisismmd-classification-eval

Analysis of Social Media Data using Multimodal Deep Learning for Disaster Response — Ferda Offli, Firoj Alam, Muhammad Imran (2020) (arXiv:2004.11838, 2020)

What this evaluates

Evaluates multimodal deep learning models for disaster response by classifying social media posts into informativeness categories (informative vs. not-informative) and humanitarian content categories (e.g., affected individuals, rescue efforts, infrastructure damage). It tests the model's ability to jointly learn from text and image modalities to improve classification performance over unimodal baselines.

Datasets

• CrisisMMD — total ?; splits: train (-1), dev (-1), test (-1)

Metrics

• F1-score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall. The paper reports aggregate F1 scores per task but does not specify whether macro- or micro-averaging is used across classes.
• accuracy — range: [0, 1]
  • Ratio of correctly predicted labels to the total number of instances.

Input / output format

Input: Tweet text (preprocessed by removing stop words, non-ASCII characters, numbers, URLs, hashtags, and replacing punctuation with spaces) and/or tweet image (pixels scaled to 0-1 and normalized per ImageNet). Multimodal input concatenates feature vectors from both modalities.

Output: Classification label: 'Informative' or 'Not-informative' for the informativeness task; one of five categories ('Affected individuals', 'Rescue volunteering or donation effort', 'Infrastructure and utility damage', 'Other relevant information', 'Not-humanitarian') for the humanitarian task.

Scoring recipe

def compute_metrics(predictions, gold_labels):
    from sklearn.metrics import f1_score, accuracy_score
    # Paper does not specify averaging; macro-average is standard
    f1 = f1_score(gold_labels, predictions, average='macro')
    acc = accuracy_score(gold_labels, predictions)
    return {'f1': f1, 'accuracy': acc}

Common pitfalls

• The dataset contains unpaired text and image instances (different counts for text vs. image in each split), so models must handle missing modalities gracefully.
• The paper does not specify whether F1 is macro- or micro-averaged across the 5 humanitarian classes.
• Early stopping and learning rate scheduling are tuned on the development set, which may lead to optimistic test performance given the small dev split (15%).

Evidence (verbatim from paper)

To measure the performance of the trained models, we use several well-known metrics such as accuracy, precision, recall, and F1-score.

Citation

@misc{offli2020analysis,
  title={Analysis of Social Media Data using Multimodal Deep Learning for Disaster Response},
  author={Ferda Offli, Firoj Alam, Muhammad Imran (2020)},
  year={2020},
  note={arXiv:2004.11838}
}

• arXiv: 2004.11838