Lightweight Action Recognition Eval

lightweight-action-recognition-eval

Evaluating Transformers for Lightweight Action Recognition — Koot et al. (2021) (arXiv:2111.09641, 2021)

What this evaluates

Evaluates the real-world efficiency (training and inference latency, VRAM footprint) of video action recognition models across desktop GPUs and mobile devices, alongside their classification accuracy on standard benchmarks.

Datasets

• EK100 — total 67217; splits: train (67217), val (9668), test (3568)
• SSV2 — total 168913; splits: train (168913), val (24777), test (174)
• K400 — total 246535; splits: train (246535), val (19907), test (400)

Metrics

• relative latency (primary) — range: other
  • Measured as the ratio of a model's average runtime to the fastest model's runtime on the same device, where the fastest model is normalized to 1.0. Measured for training steps, batch-mode inference, and single-instance inference.
• VRAM requirement — range: other
  • Peak video memory usage in gigabytes (GB) during training, batch-mode inference, and single-instance inference, measured on a representative set of desktop GPUs and mobile devices.

Input / output format

Input: Video clips resized to 224x224 frames, processed in batches (size 8 for training, 32 for batch inference, 1 for single-instance) or as single instances.

Output: Predicted action class labels, plus per-model per-device latency (relative or absolute) and peak VRAM usage (GB).

Scoring recipe

# Measure raw latencies and VRAM
latencies = {m: measure_avg_latency(m, device, batch_size) for m in models}
vram = {m: measure_peak_vram(m, device, batch_size) for m in models}

# Normalize latency relative to fastest model on each device
fastest = min(latencies.values())
relative_latency = {m: l / fastest for m, l in latencies.items()}

Common pitfalls

• Relative latency is normalized per device, making cross-device comparisons non-trivial.
• Mobile inference latency varies drastically across accelerators (CPU, GPU, ANE) and is less consistent than desktop.
• Swin and TSM are excluded from mobile benchmarks due to lack of mobile-compatible implementations.

Evidence (verbatim from paper)

Instead, we measure the latency of all models on 5 desktop GPUs and 5 mobile devices. ... We use mixed precision and take the mean latency over 100 runs, excluding 10 warm-up runs. Because each GPU yields different speeds, we report relative latency only, where the fastest model has a latency of 1.0. ... Besides latency, the VRAM requirement of a model is one of the biggest constraints during training... Table 2 gives an overview of how much VRAM each model requires.

Citation

@misc{koot2021evaluatingtransformers,
  title={Evaluating Transformers for Lightweight Action Recognition},
  author={Koot et al. (2021)},
  year={2021},
  note={arXiv:2111.09641}
}

• arXiv: 2111.09641