Har Wearable Sensor Eval

har-wearable-sensor-eval

Deep, Convolutional, and Recurrent Models for Human Activity Recognition using Wearables — Hammerla et al. (2016) (arXiv:1604.08880, 2016)

What this evaluates

Evaluates deep learning architectures (CNNs, LSTMs, DNNs) for frame-by-frame human activity recognition using wearable sensor time-series data. It probes the models' capacity to capture temporal dependencies and generalize across diverse domains (kitchen gestures, lifestyle/exercise, and medical gait analysis) while handling severe class imbalance.

Datasets

• Opportunity — total ?; splits: train (-1), val (-1), test (-1)
• PAMAP2 — total ?; splits: train (-1), val (-1), test (-1)
• Daphnet Gait — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• mean f1-score (primary) — range: [0, 1]
  • Unbiased average of per-class F1 scores: $F_{m}=\frac{2}{|c|}\sum_{c}\frac{\text{prec}{c}\times\text{recall}{c}}{\text{prec}{c}+\text{recall}{c}}$. Computed independently of class distribution to handle dataset bias.
• weighted f1-score — range: [0, 1]
  • Class-proportion-weighted average of per-class F1 scores: $F_{w}=2\sum_{c}\frac{N_{c}}{N_{\text{total}}}\frac{\text{prec}{c}\times\text{recall}{c}}{\text{prec}{c}+\text{recall}{c}}$, where $N_{c}$ is samples in class $c$ and $N_{\text{total}}$ is total samples. Used for direct comparison with prior state-of-the-art.

Input / output format

Input: Sliding windows of multi-axis wearable sensor data (accelerometer/IMU) sampled at dataset-specific frequencies (30Hz, 33.3Hz, or 32Hz). Window duration and overlap vary by dataset (e.g., 1s/50% overlap for Opportunity and Daphnet Gait; 5.12s/78% overlap for PAMAP2).

Output: Frame-by-frame activity or gesture classification labels corresponding to the input window.

Scoring recipe

def compute_mean_f1(predictions, gold, num_classes):
    precisions = []
    recalls = []
    for c in range(num_classes):
        tp = sum(p == c and g == c for p, g in zip(predictions, gold))
        fp = sum(p == c and g != c for p, g in zip(predictions, gold))
        fn = sum(p != c and g == c for p, g in zip(predictions, gold))
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
        precisions.append(prec)
        recalls.append(rec)
    f1_scores = [2 * p * r / (p + r) if (p + r) > 0 else 0.0 for p, r in zip(precisions, recalls)]
    return sum(f1_scores) / num_classes

Common pitfalls

• Datasets are highly imbalanced; using standard accuracy instead of mean F1 will severely misrepresent model performance.
• Splits are strictly subject-dependent (e.g., runs 4-5 from subjects 2-3 for Opportunity test). Random or subject-independent splits will invalidate comparisons with reported baselines.
• Windowing parameters (duration and overlap) differ across datasets and must be replicated exactly to match reported frame counts and temporal resolution.

Evidence (verbatim from paper)

As the datasets utilised in this work are highly biased we require performance metrics that are independent of the class distribution. We opted to estimate the mean f1-score: $F_{m}=\frac{2}{|c|}\sum_{c}\frac{\text{prec}{c}\times\text{recall}{c}}{\text{prec}{c}+\text{recall}{c}}$ (1) Related work has previously used the weighted f1-score as primary performance metric (for Opportunity). In order to compare our results to the state-of-the-art we estimate the weighted f1-score: $F_{w}=2\sum_{c}\frac{N_{c}}{N_{\text{total}}}\frac{\text{prec}{c}\times\text{recall}{c}}{\text{prec}{c}+\text{recall}{c}},$ (2) where $N_{c}$ is the number of samples in class $c$, and $N_{\text{total}}$ is the total number of samples.

Citation

@misc{hammerla2016deep,
  title={Deep, Convolutional, and Recurrent Models for Human Activity Recognition using Wearables},
  author={Hammerla et al. (2016)},
  year={2016},
  note={arXiv:1604.08880}
}

• arXiv: 1604.08880