Simmotion Retrieval Eval

simmotion-retrieval-eval

SemanticMoments: Training-Free Motion Similarity via Third Moment Features — Huberman et al. (2026) (arXiv:2602.09146, 2026)

What this evaluates

This benchmark evaluates a model's ability to retrieve videos based on semantic motion similarity, disentangling dynamic behavior from static appearance, camera viewpoint, and scene context. It tests robustness to appearance variations in controlled synthetic settings and unsynchronized, in-the-wild video pairs.

Datasets

• SimMotion-Synthetic — total ?; splits: test (-1)
• SimMotion-Real-1K — total 1000; splits: test (1000)
• Jester — total ?; splits: val (-1)

Metrics

• Retrieval accuracy (primary) — range: percent
  • The success rate of closest video retrieval, calculated as the fraction of queries for which the motion-preserving positive video is ranked first among all candidates.
• Top-1 majority vote accuracy — range: percent
  • For each query, retrieve the K=20 nearest neighbors in the embedding space. Predict the gesture label as the most frequent label among the neighbors.
• Weighted kNN accuracy — range: percent
  • Retrieve K=20 nearest neighbors and predict the label by weighting each neighbor's vote by its cosine similarity to the query.
• Top-5 weighted kNN accuracy — range: percent
  • Same as weighted kNN but considering the top-5 neighbors for prediction.

Input / output format

Input: Video frames or pre-extracted patch-level embeddings from a pretrained encoder (e.g., DINOv2, VideoMAE). For the proposed method, patch embeddings are aggregated over time to compute temporal moments (mean, variance, skewness).

Output: A ranked list of candidate videos sorted by cosine similarity to the query video embedding.

Scoring recipe

def compute_retrieval_accuracy(predictions, gold_indices):
    correct = sum(1 for pred, gold in zip(predictions, gold_indices) if pred == gold)
    return correct / len(predictions)

def compute_knn_accuracy(embeddings, labels, k=20, weighted=True):
    # Compute cosine similarity between query and all embeddings
    # Retrieve top-k indices
    # Predict label by majority vote or similarity-weighted vote
    # Return accuracy over validation set
    pass

Common pitfalls

• Confusing motion similarity with appearance or semantic similarity; models often retrieve videos with matching backgrounds or objects but different dynamics.
• Assuming optical flow or geometric alignment is required; the benchmark explicitly tests unsynchronized, real-world clips where flow consistency breaks despite similar semantics.
• Using frame-level global embeddings instead of patch-level features; the protocol shows patch-level granularity significantly improves motion structure preservation.

Evidence (verbatim from paper)

Table 1. Synthetic motion-similarity on SimMotion-Synthetic. Retrieval accuracy (higher is better) across motion-preserving edit categories. The benchmark holds motion fixed while varying appearance factors (object identity/attributes, view, and scene style), exposing where representations over-index on appearance.

Citation

@misc{huberman2026semanticmoments,
  title={SemanticMoments: Training-Free Motion Similarity via Third Moment Features},
  author={Huberman et al. (2026)},
  year={2026},
  note={arXiv:2602.09146}
}

• arXiv: 2602.09146