Weak Annotation Har Eval

weak-annotation-har-eval

Weak-Annotation of HAR Datasets using Vision Foundation Models — Bock et al. (2024) (arXiv:2408.05169, 2024)

What this evaluates

Evaluates inertial-based activity recognition models trained on weakly-supervised labels generated via vision foundation model clustering, benchmarked against fully-supervised and few-shot baselines.

Datasets

• WEAR — total ?; splits: test (-1)
• Wetlab — total ?; splits: test (-1)
• ActionSense — total ?; splits: test (-1)

Metrics

• Acc (primary) — range: percent
  • Fraction of correctly predicted activity labels out of total instances.
• F1 — range: percent
  • Macro-averaged F1 score across all activity classes, including the NULL class.

Input / output format

Input: 4-second IMU sensor data clips with 3-second overlap.

Output: Activity class label (including NULL-class).

Scoring recipe

def compute_metrics(preds, golds):
    acc = sum(p == g for p, g in zip(preds, golds)) / len(golds)
    classes = set(golds)
    f1_scores = []
    for c in classes:
        tp = sum(1 for p, g in zip(preds, golds) if p == c and g == c)
        fp = sum(1 for p, g in zip(preds, golds) if p == c and g != c)
        fn = sum(1 for p, g in zip(preds, golds) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1_scores.append(f1)
    f1 = sum(f1_scores) / len(f1_scores)
    return acc, f1

Common pitfalls

• NULL-class has low intra-class similarity and is frequently misclassified as other activities.
• ActionSense dataset labels are vision-centric, making IMU-only classification inherently difficult.
• Distance thresholding removes 50–90% of training data, drastically reducing sample size.

Evidence (verbatim from paper)

Table 3 provides an overview across the eight evaluated training scenarios. Our proposed weakly-supervised training is not only capable of outperforming the few-shot training using only the annotated centroid clips, but for the case of applying 100 clusters is close to matching accuracy scores of a fully-supervised training across all three benchmark datasets, for both inertial-based architectures.

Citation

@misc{bock2024weakannotation,
  title={Weak-Annotation of HAR Datasets using Vision Foundation Models},
  author={Bock et al. (2024)},
  year={2024},
  note={arXiv:2408.05169}
}

• arXiv: 2408.05169