Har Multimodal Classification Eval

har-multimodal-classification-eval

MuJo: Multimodal Joint Feature Space Learning for Human Activity Recognition — Fritsch et al. (2024) (arXiv:2406.03857, 2024)

What this evaluates

Probes the ability of multimodal and unimodal models to recognize human activities from wearable sensor, pose, and video data. It evaluates classification performance, data efficiency across low-data regimes (1-100% training fractions), and zero-shot transfer capability to unseen real-world datasets.

Datasets

• MM-Fit — total ?; splits: train (-1), test (-1)
• MHEALTH — total ?; splits: train (-1), test (-1)
• MyoGym — total ?; splits: train (-1), test (-1)
• MotionSense — total ?; splits: train (-1), test (-1)

Metrics

• Macro F1-Score (primary) — range: [0, 1]
  • Unweighted mean of recall per class. Calculated as the average of per-class F1 scores across all activity classes.
• Top-k Accuracy — range: [0, 1]
  • Proportion of instances where the true class appears in the top-k predicted probabilities or ranks.

Input / output format

Input: Windowed sensor data (accelerometer), 3D pose sequences, or pre-extracted video/text feature vectors. Multimodal inputs are formed by concatenating these feature representations.

Output: Predicted activity class label from a fixed vocabulary, optionally including a 'NULL' class for unknown/irrelevant activities.

Scoring recipe

def compute_metrics(predictions, gold_labels, k=1):
    classes = sorted(set(predictions) | set(gold_labels))
    f1_scores = []
    for c in classes:
        tp = sum(1 for p, g in zip(predictions, gold_labels) if p == c and g == c)
        fp = sum(1 for p, g in zip(predictions, gold_labels) if p == c and g != c)
        fn = sum(1 for p, g in zip(predictions, gold_labels) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1_scores.append(f1)
    macro_f1 = sum(f1_scores) / len(f1_scores)
    correct = sum(1 for p, g in zip(predictions, gold_labels) if g in p[:k])
    topk_acc = correct / len(gold_labels)
    return macro_f1, topk_acc

Common pitfalls

• Including a 'NULL' class for unknown activities significantly lowers scores across all datasets and modalities.
• Domain shift between synthetic pre-training data (FiMAD) and real-world evaluation sensors causes performance degradation.
• Pose modality mismatch due to different extraction pipelines (IMUTube vs. MM-Fit) reduces multimodal gains.

Evidence (verbatim from paper)

Table 1 shows that for our method the pre-trained models with trainable weights consistently outperform the baseline with respect to the Macro F1-Score on all datasets, regardless of whether 2% or 100% of the training data are used.

Citation

@misc{fritsch2024mujo,
  title={MuJo: Multimodal Joint Feature Space Learning for Human Activity Recognition},
  author={Fritsch et al. (2024)},
  year={2024},
  note={arXiv:2406.03857}
}

• arXiv: 2406.03857