Har Classification Eval

har-classification-eval

Benchmarking Classical, Deep, and Generative Models for Human Activity Recognition — Md Meem Hossain et al. (arXiv:2501.08471, 2025)

What this evaluates

Evaluates the ability of classical, deep learning, and generative models to accurately classify human activities from sensor data. Probes temporal pattern recognition, sensor fusion handling, and generalization across varying data complexities and sensor modalities.

Datasets

• UCI-HAR — total 10299; splits: train (7352), test (2947)
• Opportunity — total 525660; splits: train (341679), test (183981)
• PAMAP2 — total 1942874; splits: train (1554297), test (388575)
• WISDM — total 1073623; splits: train (858898), test (214725)
• Berkeley MHAD — total 2401920; splits: train (1801440), test (600480)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Ratio of correctly predicted activity labels to the total number of instances. Computed as (True Positives + True Negatives) / Total Instances.
• Precision — range: [0, 1]
  • Ratio of correctly predicted positive activity labels to all instances predicted as positive. Macro-averaged across classes.
• Recall — range: [0, 1]
  • Ratio of correctly predicted positive activity labels to all actual positive instances. Macro-averaged across classes.
• F1-Score — range: [0, 1]
  • Harmonic mean of Precision and Recall. Macro-averaged across classes to balance performance across all activity types.

Input / output format

Input: Time-series sensor readings (e.g., accelerometer, gyroscope) corresponding to fixed-length windows or sequences, formatted according to each dataset's standard schema.

Output: Categorical activity label (e.g., 'WALKING', 'SITTING', 'LAYING') representing the ground-truth class for the input sequence.

Scoring recipe

def compute_metrics(y_true, y_pred):
    accuracy = (y_true == y_pred).mean()
    precision = precision_score(y_true, y_pred, average='macro')
    recall = recall_score(y_true, y_pred, average='macro')
    f1 = f1_score(y_true, y_pred, average='macro')
    return {'accuracy': accuracy, 'precision': precision, 'recall': recall, 'f1_score': f1}

Common pitfalls

• Fixed train/test splits are used without cross-validation, risking split-dependent variance and over-optimistic results.
• Similar postures (e.g., SITTING vs STANDING) cause high confusion, indicating models may rely on superficial features rather than robust temporal patterns.
• Perfect scores on training data (e.g., PAMAP2) signal severe overfitting, which the authors explicitly note does not guarantee generalization to unseen data.

Evidence (verbatim from paper)

In this section, we provide the findings from our comparative analysis of the machine learning models used for HAR across the selected benchmark datasets. We thoroughly analyse the performance results, focusing on key metrics such as accuracy, precision, recall, and F1-score, which are critical for determining model success in real-world scenarios.

Citation

@misc{hossain2025benchmarking,
  title={Benchmarking Classical, Deep, and Generative Models for Human Activity Recognition},
  author={Md Meem Hossain et al.},
  year={2025},
  note={arXiv:2501.08471}
}

• arXiv: 2501.08471