Pushupbench Eval

pushupbench-eval

PushupBench: Your VLM is not good at counting pushups — Li et al. (2026) (arXiv:2604.23407, 2026)

What this evaluates

Evaluates video-language models on long-form repetition counting and temporal reasoning. It probes whether models can accurately track state changes and count actions across extended video clips, revealing weaknesses in spatio-temporal tracking compared to supervised baselines.

Datasets

• PushupBench — total 446; splits: test (446), train (391)

Metrics

• Exact Match (primary) — range: percent
  • Percentage of predictions that exactly equal the ground truth count.
• MAE — range: other
  • Mean absolute error between predicted and ground truth counts. Computed only on samples where |prediction - ground truth| ≤ 50 to exclude extreme outliers.
• R^2 — range: other
  • Coefficient of determination measuring variance explained beyond the mean baseline. Computed as 1 - (SS_res / SS_tot) on samples with |error| ≤ 50. Negative values indicate constant or random output.

Input / output format

Input: Video clips sampled at 5 fps and capped at 112 frames (uniformly spaced for longer videos), resized to 360p. Each clip is paired with one of 10 diverse prompt templates requesting the repetition count.

Output: A single integer representing the predicted number of repetitions.

Scoring recipe

def compute_metrics(predictions, ground_truths):
    exact = sum(1 for p, gt in zip(predictions, ground_truths) if p == gt) / len(predictions)
    valid = [(p, gt) for p, gt in zip(predictions, ground_truths) if abs(p - gt) <= 50]
    mae = sum(abs(p - gt) for p, gt in valid) / len(valid) if valid else 0
    y = [gt for _, gt in valid]
    y_hat = [p for p, _ in valid]
    y_mean = sum(y) / len(y)
    ss_res = sum((yi - yhati)**2 for yi, yhati in zip(y, y_hat))
    ss_tot = sum((yi - y_mean)**2 for yi in y)
    r2 = 1 - (ss_res / ss_tot) if ss_tot != 0 else 0
    return exact, mae, r2

Common pitfalls

• Ambiguous action boundaries in ~6.1% of samples require accepting multiple valid ground truth counts rather than a single rigid label.
• Extreme outliers from parsing failures or hallucinations drastically skew MAE and R^2, so the protocol explicitly excludes predictions with |error| > 50.
• Fast repetitions (<3 frames/rep at 5fps) become temporally unresolvable, causing noise in training and evaluation if sampling rates are not adjusted.

Evidence (verbatim from paper)

We report three metrics: Exact Match (percentage of predictions equaling ground truth), MAE (mean absolute error), and $R^{2}$ (coefficient of determination). For 27 samples (6.1%) with ambiguous action boundaries, we accept multiple valid ground truth counts (see AppendixB). MAE and $R^{2}$ are computed excluding predictions with $|\text{error}|>50$, as extreme outliers from parsing failures or hallucinations can drastically skew these metrics.

Citation

@misc{li2026pushupbench,
  title={PushupBench: Your VLM is not good at counting pushups},
  author={Li et al. (2026)},
  year={2026},
  note={arXiv:2604.23407}
}

• arXiv: 2604.23407