Vos Language Referring

vos-language-referring

Video Object Segmentation with Language Referring Expressions — Khoreva et al. (2018) (arXiv:1803.08006, 2018)

What this evaluates

Evaluates a model's ability to perform pixel-level video object segmentation guided by natural language referring expressions, testing both language grounding and temporal consistency in dynamic scenes.

Datasets

• DAVIS-16 — total ?; splits: train (-1), val (-1), test (-1)
• DAVIS-17 — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• performance score (primary) — range: percent
  • Not explicitly defined in the provided text; standard video object segmentation metrics (e.g., J&F or mIoU) are implied for comparing against mask-based and scribble-based baselines.

Input / output format

Input: A video sequence V={f_1,...,f_N} and a natural language referring expression Q describing the target object.

Output: A pixel-level segmentation mask for the target object in every frame where it appears.

Scoring recipe

# Not explicitly provided in text.
# Standard VOS evaluation typically follows:
for frame in video:
    pred_mask = model.predict(frame, query)
    gt_mask = get_ground_truth(frame)
    iou = intersection(pred_mask, gt_mask) / union(pred_mask, gt_mask)
    scores.append(iou)
final_score = mean(scores)

Common pitfalls

• Using image-only grounding models per frame leads to temporally inconsistent and jittery bounding box predictions.
• Training on static images (MSRA) and testing on video can cause domain shift, requiring careful augmentation (e.g., synthesized optical flow, box jittering).

Evidence (verbatim from paper)

demonstrating that language supervision achieves performance comparable to mask-based methods on DAVIS-16 and matches scribble-based approaches on the challenging DAVIS-17 dataset

Citation

@misc{khoreva2018voslanguage,
  title={Video Object Segmentation with Language Referring Expressions},
  author={Khoreva et al. (2018)},
  year={2018},
  note={arXiv:1803.08006}
}

• arXiv: 1803.08006