Stvg Eval

stvg-eval

Bridging Time and Space: Decoupled Spatio-Temporal Alignment for Video Grounding — Tu et al. (2026) (arXiv:2604.08014, 2026)

What this evaluates

Evaluates a model's ability to localize a specific object or event in a video based on a natural language query. It measures both temporal localization (identifying the correct start and end timestamps) and spatial localization (predicting accurate bounding box trajectories across the video frames).

Datasets

• VidSTG — total 10000; splits: train (-1), val (-1), test (-1)
• HCSTVG-v1&v2 — total 107000; splits: train (-1), val (-1), test (-1)

Metrics

• m_tIoU — range: [0, 1]
  • Mean temporal Intersection-over-Union (tIoU) computed as the overlap between predicted and ground-truth temporal intervals divided by their union, averaged across all queries.
• m_vIoU (primary) — range: [0, 1]
  • Mean 3D Intersection-over-Union (vIoU) computed as the average spatial overlap between predicted and ground-tr bounding box trajectories across all frames in the temporal interval, averaged across all queries.
• vIoU@0.3 — range: [0, 1]
  • Proportion of samples where the predicted vIoU exceeds a threshold of 0.3, indicating performance under moderate localization demands.
• vIoU@0.5 — range: [0, 1]
  • Proportion of samples where the predicted vIoU exceeds a threshold of 0.5, indicating performance under strict localization demands.

Input / output format

Input: Video frames uniformly sampled at 2 FPS, paired with a natural language query (declarative or interrogative sentence).

Output: Predicted temporal interval (start and end timestamps) and a sequence of spatial bounding boxes (x, y, width, height) for each frame within the interval.

Scoring recipe

def compute_stvg_metrics(predictions, gold):
    tIoU_scores = []
    vIoU_scores = []
    for pred, gt in zip(predictions, gold):
        t_iou = intersection_union(pred.t_start, pred.t_end, gt.t_start, gt.t_end)
        tIoU_scores.append(t_iou)
        v_iou = mean_3d_iou(pred.boxes, gt.boxes)
        vIoU_scores.append(v_iou)
    m_tIoU = sum(tIoU_scores) / len(tIoU_scores)
    m_vIoU = sum(vIoU_scores) / len(vIoU_scores)
    vIoU_03 = sum(1 for v in vIoU_scores if v > 0.3) / len(vIoU_scores)
    vIoU_05 = sum(1 for v in vIoU_scores if v > 0.5) / len(vIoU_scores)
    return m_tIoU, m_vIoU, vIoU_03, vIoU_05

Common pitfalls

• Failing to separate declarative and interrogative sentence subsets when reporting VidSTG results, as they require different reasoning strategies and yield significantly different scores.
• Ignoring the 2 FPS video sampling rate during inference, which disrupts the temporal alignment and spatial token processing pipeline.
• Mixing training data (HCSTVG-v1&v2, synthetic ReVOS data) with test sets without strict train-test separation, leading to inflated performance.

Evidence (verbatim from paper)

Following (Gu et al., 2024, 2025a; Jin et al., 2022; Su et al., 2021; Yang et al., 2022), m_tIoU, m_vIoU, vIoU@R are adopted as evaluation metrics for STVG. m_tIoU reports the mean temporal Intersection-over-Union (tIoU) of predicted versus ground-truth intervals, evaluating temporal grounding. m_vIoU computes the average 3D IoU of spatio-temporal tubes to assess spatial grounding. vIoU@R measures the proportion of samples with vIoU exceeding a threshold R (e.g., 0.3, 0.5), indicating performance under precise localization demands.

Citation

@misc{tu2026bridging,
  title={Bridging Time and Space: Decoupled Spatio-Temporal Alignment for Video Grounding},
  author={Tu et al. (2026)},
  year={2026},
  note={arXiv:2604.08014}
}

• arXiv: 2604.08014