Medic Eval

medic-eval

MEDIC: A Multi-Task Learning Dataset for Disaster Image Classification — Alam et al. (2021) (arXiv:2108.12828, 2021)

What this evaluates

Evaluates image classification models on disaster-related social media images across four interdependent tasks: disaster type, informativeness, humanitarian relevance, and damage severity. It tests both single-task and multi-task learning capabilities, including multiclass and multilabel classification settings.

Datasets

• MEDIC — total 71198; splits: train (-1), dev (-1), test (-1); repo https://github.com/firojalam/medic

Metrics

• weighted F1-score (primary) — range: percent
  • Weighted average of per-class F1-scores, where each class's F1 is weighted by its support (number of true instances in the dataset).
• micro F1-score — range: percent
  • Micro-averaged F1 computed by aggregating true positives, false positives, and false negatives across all classes before calculating precision, recall, and F1.
• humming loss — range: other
  • A loss function for multilabel classification that penalizes incorrect label assignments based on the Hamming distance between predicted and true label sets.

Input / output format

Input: Raw social media images paired with ground-truth annotations for up to four tasks: disaster type (multiclass), informativeness (binary), humanitarian relevance (multiclass), and damage severity (multiclass). Multilabel variants also provide multiple active labels per task.

Output: Predicted class labels or probability distributions for each task. Multiclass tasks use softmax outputs; multilabel tasks use sigmoid outputs.

Scoring recipe

def compute_weighted_f1(y_true, y_pred, classes):
    f1s = []
    for c in classes:
        tp = sum(1 for yt, yp in zip(y_true, y_pred) if yt == c and yp == c)
        fp = sum(1 for yt, yp in zip(y_true, y_pred) if yt != c and yp == c)
        fn = sum(1 for yt, yp in zip(y_true, y_pred) if yt == c and yp != c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1s.append((f1, sum(1 for yt in y_true if yt == c)))
    return sum(f * w for f, w in f1s) / sum(w for _, w in f1s)

Common pitfalls

• The dataset exhibits class imbalance, but the authors note that the majority-class baseline performs similarly across tasks, suggesting imbalance does not drastically skew results.
• Multilabel metrics (micro F1, humming loss) are reported separately from multiclass metrics (weighted F1), making direct cross-setting comparisons invalid.
• Multi-task learning experiments use equal task weights (w_i=1) and hard parameter sharing, which may not reflect optimal task balancing strategies.

Evidence (verbatim from paper)

To measure the performance of each classifier and for each task setting, we use weighted average precision (P), recall (R), and F1-score (F1), which are widely used in the literature. For the multilabel experiments we computed micro average precision (P), recall (R), F1-score (F1) and humming loss, which are commonly used metrics [89, 90].

Citation

@misc{alam2021medic,
  title={MEDIC: A Multi-Task Learning Dataset for Disaster Image Classification},
  author={Alam et al. (2021)},
  year={2021},
  note={arXiv:2108.12828}
}

• arXiv: 2108.12828