Mosev2 Eval

mosev2-eval

MOSEv2: A More Challenging Dataset for Video Object Segmentation in Complex Scenes — Ding et al. (2025) (arXiv:2508.05630, 2025)

What this evaluates

This benchmark evaluates video object segmentation and tracking models under highly complex, unconstrained real-world conditions. It specifically probes robustness to severe occlusions, frequent object disappearance and reappearance, adverse weather, low-light environments, camouflage, and knowledge-dependent scenarios.

Datasets

• MOSEv2 — total 4713; splits: train (3666), val (433), test (614)

Metrics

• $\mathcal{J}\\&\dot{\mathcal{F}}$ (primary) — range: percent
  • The mean of the Jaccard index ($\mathcal{J}$) and the adaptive F-score ($\dot{\mathcal{F}}$). The adaptive F-score uses a dynamic threshold to better assess boundary quality across objects of varying sizes, yielding stricter scores than the standard F-score.
• $\mathcal{J}\\&\dot{\mathcal{F}}_{d}$ — range: percent
  • J&F dot computed only on frames where the target object is disappearing or occluded.
• $\mathcal{J}\\&\dot{\mathcal{F}}_{r}$ — range: percent
  • J&F dot computed only on frames where the target object reappears after being absent.

Input / output format

Input: For semi-supervised VOS: a video sequence plus an initial prompt on the first frame (full mask, bounding box, or point coordinates). For unsupervised VOS: video sequence only (with a proposal limit of 20). For interactive VOS: video sequence plus iterative user clicks.

Output: Per-frame binary segmentation masks (or instance masks) for the target object(s) across the entire video sequence.

Scoring recipe

def compute_jf_dot(pred_masks, gt_masks):
    j_scores = [jaccard(p, g) for p, g in zip(pred_masks, gt_masks)]
    f_scores = [adaptive_fscore(p, g) for p, g in zip(pred_masks, gt_masks)]
    jf_scores = [(j + f) / 2 for j, f in zip(j_scores, f_scores)]
    return sum(jf_scores) / len(jf_scores)

Common pitfalls

• Using the standard F-score ($\mathcal{F}$) instead of the adaptive F-score ($\dot{\mathcal{F}}$), which overestimates boundary accuracy for small or irregular objects.
• Failing to limit unsupervised methods to a maximum of 20 object proposals per video, leading to unfair performance inflation.
• Confusing disappearance ($\mathcal{J}\&\dot{\mathcal{F}}{d}$) and reappearance ($\mathcal{J}\&\dot{\mathcal{F}}{r}$) sub-metrics, which require strict temporal masking of frames where the target is absent vs. present.

Evidence (verbatim from paper)

We evaluate model performance using standard metrics ($\mathcal{J}$, $\mathcal{F}$, and $\mathcal{J}\&\mathcal{F}$) on MOSEv2 validation set, following the DAVIS protocol*[[2], [3]]*. To better capture the complex challenges in MOSEv2, we additionally report $\dot{\mathcal{F}}$, $\mathcal{J}\&\dot{\mathcal{F}}$, $\mathcal{J}\&\dot{\mathcal{F}}{d}$, and $\mathcal{J}\&\dot{\mathcal{F}}{r}$ as described in [Section 3.3]. Among them, $\mathcal{J}\&\dot{\mathcal{F}}$ is selected as the primary evaluation metric.

Citation

@misc{ding2025mosev2,
  title={MOSEv2: A More Challenging Dataset for Video Object Segmentation in Complex Scenes},
  author={Ding et al. (2025)},
  year={2025},
  note={arXiv:2508.05630}
}

• arXiv: 2508.05630