Wake Vision Eval

wake-vision-eval

Wake Vision: A Tailored Dataset and Benchmark Suite for TinyML Computer Vision Applications — Banbury et al. (2024) (arXiv:2405.00892, 2024)

What this evaluates

Evaluates the robustness and accuracy of TinyML person detection models across diverse demographic, environmental, and visual conditions. It benchmarks binary classification performance on large-scale, real-world image datasets tailored for resource-constrained devices.

Datasets

• Wake Vision — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• accuracy (primary) — range: percent
  • Top-level test accuracy calculated as the percentage of correctly classified binary labels on the held-out test set.
• F1 score — range: [0, 1]
  • Harmonic mean of precision and recall, computed per fine-grained subgroup (e.g., age, lighting, distance) and averaged across three models per dataset.

Input / output format

Input: 224x224x3 RGB images representing real-world scenes for binary person detection.

Output: Binary classification label (person vs. non-person/wake word).

Scoring recipe

def compute_accuracy(preds, gold):
    return sum(p == g for p, g in zip(preds, gold)) / len(gold) * 100

def compute_f1(preds, gold):
    tp = sum(p == 1 and g == 1 for p, g in zip(preds, gold))
    fp = sum(p == 1 and g == 0 for p, g in zip(preds, gold))
    fn = sum(p == 0 and g == 1 for p, g in zip(preds, gold))
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

Common pitfalls

• Models trained on VWW may outperform Wake Vision models on the VWW test set due to domain shift, not superior generalization.
• Fine-grained F1 scores are averaged across three models per dataset, not reported per individual model.

Evidence (verbatim from paper)

The Wake Vision model exhibits superior robustness across the challenging settings exercised by our benchmarking suite. For instance, on the “Depictions” benchmark, which evaluates performance on images containing persons, non-person objects, or no depictions, the Wake Vision model achieves an F1 score of 0.71 for person depictions, outperforming the VWW model’s 0.66.

Citation

@misc{banbury2024wakevision,
  title={Wake Vision: A Tailored Dataset and Benchmark Suite for TinyML Computer Vision Applications},
  author={Banbury et al. (2024)},
  year={2024},
  note={arXiv:2405.00892}
}

• arXiv: 2405.00892