Track Any State Eval

track-any-state-eval

Tracking and Understanding Object Transformations — Sun et al. (2025) (arXiv:2511.04678, 2025)

What this evaluates

Evaluates a model's ability to track objects through appearance-changing state transformations and explicitly model those transformations as a state graph. It probes spatiotemporal continuity, zero-shot object recovery, and semantic reasoning about object interactions.

Datasets

• VOST — total 70; splits: val (70)
• VSCOS — total 98; splits: val (98)
• M3-VOS — total 479; splits: test (479)
• DAVIS 2017 — total ?; splits: val (-1)
• VOST-TAS — total 57; splits: test (57)

Metrics

• Jaccard (J) (primary) — range: [0, 1]
  • Mean Intersection-over-Union (IoU) between predicted and ground-truth masks across all frames and objects.
• J_tr — range: [0, 1]
  • Jaccard score computed only over the last 25% of frames in each video.
• Precision (P) — range: [0, 1]
  • Mean IoU for predicted masks that achieve IoU > 0.5 with any ground-truth mask.
• Recall (R) — range: [0, 1]
  • Mean IoU for ground-truth masks that achieve IoU > 0.5 with any predicted mask.
• Temporal Precision (T_P) — range: [0, 1]
  • Precision of predicted transformation start/end frames against annotated temporal boundaries.
• Temporal Recall (T_R) — range: [0, 1]
  • Recall of predicted transformation start/end frames against annotated temporal boundaries.
• Semantic Accuracy (S_V) — range: [0, 1]
  • Accuracy of predicted action verbs, computed only on transformations that are correctly localized temporally.
• Semantic Accuracy (S_O) — range: [0, 1]
  • Accuracy of predicted resulting object descriptions, computed only on objects matched with IoU > 0.5.
• Spatiotemporal Recall (H_ST) — range: [0, 1]
  • Recall of transformations where temporal localization is correct AND all resulting objects are matched with IoU > 0.5.
• Overall Recall (H) — range: [0, 1]
  • Recall of transformations where temporal localization is correct, all resulting objects are matched with IoU > 0.5, AND all action/object descriptions are correct.

Input / output format

Input: Video frames with an initial object mask or tracklet prompt.

Output: Per-frame segmentation masks for the tracked object(s), plus a structured state graph listing each transformation with start/end frames, an action verb, and resulting objects with masks and text descriptions.

Scoring recipe

def compute_tracking(pred_masks, gt_masks):
    ious = compute_iou(pred_masks, gt_masks) # per frame
    J = mean(ious)
    J_tr = mean(ious[last_25_frames])
    P = mean(max_iou_per_pred > 0.5)
    R = mean(max_iou_per_gt > 0.5)
    return J, J_tr, P, R

def compute_state_graph(pred_trans, gt_trans):
    T_P = precision(pred_trans.start_end, gt_trans.start_end)
    T_R = recall(pred_trans.start_end, gt_trans.start_end)
    S_V = accuracy(pred_verbs[correct_temporal], gt_verbs[correct_temporal])
    S_O = accuracy(pred_obj_descs[iou>0.5], gt_obj_descs[iou>0.5])
    H_ST = recall(trans where temporal_correct AND all_objects_iou>0.5)
    H = recall(trans where temporal_correct AND all_objects_iou>0.5 AND all_descs_correct)
    return T_P, T_R, S_V, S_O, H_ST, H

Common pitfalls

• J_tr is evaluated exclusively on the final 25% of frames, not the full video duration.
• Semantic accuracy metrics (S_V, S_O) are conditional on successful temporal/spatial localization, masking overall system failure rates.
• Object matching in state graph evaluation strictly uses an IoU > 0.5 threshold, which may penalize slight segmentation drift.
• DAVIS 2017 serves as a non-transformation baseline but is evaluated with the same protocol, potentially inflating expectations for standard tracking.

Evidence (verbatim from paper)

To evaluate state graph quality, we report precision ${\mathcal{T}}{P}$ and recall ${\mathcal{T}}{R}$ for temporal localization within annotated transformation boundaries, and description accuracy for correctly localized action verbs (${\mathcal{A}}{V}$) and resulting objects (${\mathcal{A}}{O}$) with IoU $>0.5$.

Citation

@misc{sun2025tracking,
  title={Tracking and Understanding Object Transformations},
  author={Sun et al. (2025)},
  year={2025},
  note={arXiv:2511.04678}
}

• arXiv: 2511.04678