Synthref Refvos Eval

synthref-refvos-eval

SynthRef: Generation of Synthetic Referring Expressions for Object Segmentation — Kazakos et al. (2021) (arXiv:2106.04403, 2021)

What this evaluates

Evaluates the effectiveness of a synthetic referring expression dataset for training language-guided video object segmentation models. It measures segmentation accuracy when models are trained on synthetic versus human annotations and evaluated on standard referring video segmentation benchmarks.

Datasets

• DAVIS-2017 — total ?; splits: val (30), train+val (90)
• Refer-YouTube-VOS — total ?; splits: test (-1)

Metrics

• J&F (primary) — range: percent
  • Average of Jaccard (J) and F-score (F) indices measuring overlap between predicted and ground-truth segmentation masks. Defined in supplementary material.
• Prec@0.5 — range: percent
  • Percentage of frames where the Intersection over Union (IoU) between predicted and ground-truth masks exceeds 0.5.
• Prec@0.9 — range: percent
  • Percentage of frames where the IoU between predicted and ground-truth masks exceeds 0.9.
• Mean IoU — range: percent
  • Average Intersection over Union across all frames and objects in the evaluation set.

Input / output format

Input: Video sequences paired with referring expressions (text) and ground-truth segmentation masks.

Output: Per-frame binary segmentation masks for the referred object.

Scoring recipe

def compute_metrics(pred_masks, gt_masks):
    ious = [iou(p, g) for p, g in zip(pred_masks, gt_masks)]
    j = mean(ious)
    f = f_score(pred_masks, gt_masks)
    jf = (j + f) / 2
    prec_05 = mean([1 if i > 0.5 else 0 for i in ious]) * 100
    prec_09 = mean([1 if i > 0.9 else 0 for i in ious]) * 100
    mean_iou = mean(ious) * 100
    return jf, prec_05, prec_09, mean_iou

Common pitfalls

• J&F metric definition is located in the supplementary material, not the main text.
• Refer-YouTube-VOS evaluation uses only a subset of videos that correspond to the synthetic dataset, not the full benchmark.
• Performance gains vary significantly depending on whether the model is evaluated after zero-shot pretraining or after fine-tuning on target domain data.

Evidence (verbatim from paper)

DAVIS-2017 We report the gains of adding the synthetic dataset when evaluating on the standard validation partition of DAVIS-2017 (30 videos), but also on the combined training and validation partitions (90 videos), to obtain more statistically significant results. The J&F metric is defined in the supplementary material.

Citation

@misc{kazakos2021synthref,
  title={SynthRef: Generation of Synthetic Referring Expressions for Object Segmentation},
  author={Kazakos et al. (2021)},
  year={2021},
  note={arXiv:2106.04403}
}

• arXiv: 2106.04403