Har Continual Learning Eval

har-continual-learning-eval

Continual Learning in Sensor-based Human Activity Recognition: an Empirical Benchmark Analysis — Jha et al. (2021) (arXiv:2104.09396, 2021)

What this evaluates

This benchmark evaluates continual learning algorithms on sensor-based human activity recognition (HAR) datasets. It measures how well models balance plasticity (learning new activities) and stability (retaining old activities) while incrementally processing tasks, specifically probing robustness to class imbalance, sensor noise, and cross-user data leakage.

Datasets

• House A (HA) — total ?; splits: train (-1), test (-1)
• CASAS (WS, Milan, Twor, Aruba) — total ?; splits: train (-1), test (-1)
• PAMAP2 — total ?; splits: train (-1), test (-1)
• DSADS — total ?; splits: train (-1), test (-1)
• HAPT — total ?; splits: train (-1), test (-1)

Metrics

• F1-scores (primary) — range: [0, 1]
  • Standard F1 score computed per class and averaged (macro) or computed globally (micro) to handle class imbalance. Base, old, new, and all-task accuracies are all reported using these F1 variants.
• Forgetting Score (FS) — range: [0, 1]
  • FS_k = (1/(k-1)) * sum_{j=1}^{k-1} (1 - a_{k,j} / max_{l in 1..k-1} a_{l,j}), where a_{k,j} is the accuracy on old task j at current task k. Normalized to [0,1] where 1 indicates complete forgetting.

Input / output format

Input: 60-second segmented sensor windows (accelerometer or binary event-driven) with extracted features (e.g., activation ratios, mean, std, correlations, spectrum peaks).

Output: Predicted activity class label for the current incremental task.

Scoring recipe

def compute_metrics(preds, gold, old_tasks_acc, current_task_idx):
    # F1-scores (macro/micro) for current task
    f1_macro = f1_score(gold, preds, average='macro')
    f1_micro = f1_score(gold, preds, average='micro')
    
    # Forgetting Score (FS) across old tasks
    fs_values = []
    for j in old_tasks_acc:
        max_acc_j = max(old_tasks_acc[j])
        curr_acc_j = old_tasks_acc[j][-1]
        if max_acc_j > 0:
            f_j = 1 - (curr_acc_j / max_acc_j)
            fs_values.append(f_j)
    FS_k = sum(fs_values) / len(fs_values) if fs_values else 0.0
    return {'F1_macro': f1_macro, 'F1_micro': f1_micro, 'FS': FS_k}

Common pitfalls

• Splitting data by sample instead of by user, causing severe data leakage in multi-user accelerometer datasets.
• Using plain accuracy instead of F1-macro/micro, which masks performance degradation on minority classes due to HAR dataset imbalance.
• Reporting results on a single random task sequence, which introduces high variance and bias due to task ordering effects.

Evidence (verbatim from paper)

When training a new task, we compute three types of accuracy. (1) Base accuracy – the accuracy of recognising the activity classes in the first task; (2) Old accuracy – the accuracy of recognising all the old activity classes that have learnt before the current task. Both base and old accuracy will indicate the stability of the model; (3) New accuracy – the accuracy of recognising the new activity classes in the current task, which will indicate the plasticity of the model; and (4) All accuracy – the accuracy of recognising all the activity classes that have learned so far, which will indicate the overall performance of the model. The accuracy is measured in F1-scores, which balances precision and recall. As most of the HAR datasets have an imbalanced class distribution, we use F1-macro and F1-micro.

Citation

@misc{jha2021continual,
  title={Continual Learning in Sensor-based Human Activity Recognition: an Empirical Benchmark Analysis},
  author={Jha et al. (2021)},
  year={2021},
  note={arXiv:2104.09396}
}

• arXiv: 2104.09396