Har Weighted F1 Eval

har-weighted-f1-eval

Layer-wise training convolutional neural networks with smaller filters for human activity recognition using wearable sensors — Yin Tang et al. (arXiv:2005.03948, 2020)

What this evaluates

Evaluates the ability of lightweight convolutional neural networks to accurately classify human activities from wearable sensor time-series data. It probes the trade-off between model compression (parameter count and FLOPs) and classification performance on highly imbalanced, multi-class activity recognition tasks.

Datasets

• UCI-HAR — total ?; splits: train (-1), test (-1)
• OPPORTUNITY — total 650000; splits: train (-1), test (-1)
• PAMAP2 — total ?; splits: train (-1), test (-1)
• UNIMIB-SHAR — total 11771; splits: train (-1), test (-1)
• WISDM — total ?; splits: train (-1), test (-1)

Metrics

• weighted F1 score (primary) — range: percent
  • Weighted average of per-class F1 scores, where each class's F1 is weighted by its proportion of total samples: F1 = 2 * Σ (N_c / N_total) * (Precision_c * Recall_c) / (Precision_c + Recall_c).

Input / output format

Input: Fixed-width sliding windows of multi-channel time-series sensor data (accelerometer, gyroscope, or IMU readings) sampled at dataset-specific frequencies.

Output: Softmax probability distribution over activity classes, with the predicted class label selected as the final output.

Scoring recipe

def weighted_f1(predictions, gold, num_classes):
    class_counts = np.bincount(gold, minlength=num_classes)
    total = class_counts.sum()
    f1_scores = []
    for c in range(num_classes):
        tp = np.sum((predictions == c) & (gold == c))
        fp = np.sum((predictions == c) & (gold != c))
        fn = np.sum((predictions != c) & (gold == c))
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1_c = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1_scores.append((class_counts[c] / total) * f1_c)
    return sum(f1_scores) * 100

Common pitfalls

• Using overall classification accuracy instead of weighted F1, which fails on highly imbalanced datasets (e.g., OPPORTUNITY's 72% NULL class).
• Applying Lego filters to the first convolutional layer or the final fully connected layer, which the authors explicitly state should be avoided to prevent feature sharing and performance degradation.
• Reporting single-run results instead of averaging over 5 independent training runs as required by the protocol.

Evidence (verbatim from paper)

Since human activity datasets are often highly unbalanced, the overall classification accuracy is not an appropriate measure to evaluate HAR tasks. Requiring performance metrics that are independent of the class distribution, we evaluate the models using the weighted F1 score: F1=2∑(Nc/Ntotal)(Precisionc×Recallc)/(Precisionc+Recallc) which considers the correct classification of each class equally important. Nc is the number of samples in class c, and Ntotal is the total number of samples. The experiments are repeated 5 times and the mean F1 score is used as the final measure to evaluate model performance.

Citation

@misc{tang2020layerwise,
  title={Layer-wise training convolutional neural networks with smaller filters for human activity recognition using wearable sensors},
  author={Yin Tang et al.},
  year={2020},
  note={arXiv:2005.03948}
}

• arXiv: 2005.03948