Omnifall Eval

omnifall-eval

OmniFall: A Unified Staged-to-Wild Benchmark for Human Fall Detection — Schneider et al. (2025) (arXiv:2505.19889, 2025)

What this evaluates

Evaluates human fall detection and action recognition capabilities across controlled (staged) and uncontrolled (wild) video domains. It probes a model's ability to classify a 10-class activity taxonomy, detect binary fall/fallen states, and segment action timelines, while measuring generalization gaps between in-distribution and out-of-distribution settings.

Datasets

• CMDFall — total ?; splits: cross-subject (-1), cross-view (-1)
• UP-Fall — total ?; splits: cross-subject (-1), cross-view (-1)
• Le2i — total ?; splits: cross-subject (-1), cross-view (-1)
• GMDCSA24 — total ?; splits: cross-subject (-1), cross-view (-1)
• EDF — total ?; splits: cross-subject (-1), cross-view (-1)
• OCCU — total ?; splits: cross-subject (-1), cross-view (-1)
• CaucaFall — total ?; splits: cross-subject (-1), cross-view (-1)
• MCFD — total ?; splits: cross-subject (-1), cross-view (-1)
• OOPS-Fall — total ?; splits: cross-subject (-1), cross-view (-1)

Metrics

• Balanced Accuracy (primary) — range: [0, 1]
  • The average of sensitivity (recall) and specificity across classes. Computed as (TP/(TP+FN) + TN/(TN+FP)) / 2.
• Macro F1 (primary) — range: [0, 1]
  • The unweighted mean of F1-scores calculated independently for each of the 10 activity classes.
• Sensitivity — range: [0, 1]
  • True positive rate for the positive class (e.g., fall or fallen). Calculated as TP / (TP + FN).
• Specificity — range: [0, 1]
  • True negative rate for the negative class. Calculated as TN / (TN + FP).
• F1-score — range: [0, 1]
  • Harmonic mean of precision and recall for binary fall/fallen detection. Calculated as 2 * TP / (2 * TP + FP + FN).
• Segmental F1@25 — range: [0, 1]
  • Segmental F1 score computed at an Intersection-over-Union (IoU) threshold of 25% between predicted and ground-truth action segments.
• Edit Distance — range: [0, 1]
  • Normalized edit distance measuring the minimum number of insertions, deletions, and substitutions required to transform the predicted label sequence into the ground truth.

Input / output format

Input: Video sequences (raw frames or pre-extracted I3D/VideoMAE features, downsampled to 10 fps for segmentation). Features are processed as temporal sequences (e.g., 18 tokens) with sinusoidal position encodings.

Output: Class labels (10-class taxonomy or binary fall/fallen/combined) and temporal segment boundaries for timeline segmentation.

Scoring recipe

def compute_classification_metrics(y_true, y_pred, classes):
    tp = sum((y_true == c) & (y_pred == c) for c in classes)
    fp = sum((y_true != c) & (y_pred == c) for c in classes)
    fn = sum((y_true == c) & (y_pred != c) for c in classes)
    tn = sum((y_true != c) & (y_pred != c) for c in classes)
    
    sensitivity = tp / (tp + fn) if (tp + fn) > 0 else 0
    specificity = tn / (tn + fp) if (tn + fp) > 0 else 0
    f1 = 2 * tp / (2 * tp + fp + fn) if (2 * tp + fp + fn) > 0 else 0
    balanced_acc = (sensitivity + specificity) / 2
    
    return balanced_acc, f1, sensitivity, specificity

def compute_segmental_f1(pred_segments, gt_segments, iou_thresh=0.25):
    # Match predicted and ground-truth segments by IoU >= threshold
    # Compute precision, recall, and F1 over matched segments
    pass

Common pitfalls

• Confusing cross-subject (CS) and cross-view (CV) evaluation splits, which differ significantly in training corpus size and generalization behavior.
• Overlooking the staged-to-wild domain shift: models often maintain high specificity on OOPS-Fall but suffer drastic sensitivity drops due to overfitting to staged appearance cues.
• Ignoring the two-level weighting strategy (dataset balancing and label-smoothed cross-entropy) required to handle severe class and domain imbalance during training.

Evidence (verbatim from paper)

We perform separate training runs for cross-subject and cross-view evaluations, reporting balanced accuracy, overall accuracy, and macro-F1 for the 10-class task, plus sensitivity, specificity and F1-score for binary fall detection subtasks.

Citation

@misc{schneider2025omnifall,
  title={OmniFall: A Unified Staged-to-Wild Benchmark for Human Fall Detection},
  author={Schneider et al. (2025)},
  year={2025},
  note={arXiv:2505.19889}
}

• arXiv: 2505.19889