Chatterbox Mrg Eval

chatterbox-mrg-eval

ChatterBox: Multi-round Multimodal Referring and Grounding — Tian et al. (2024) (arXiv:2401.13307, 2024)

What this evaluates

Evaluates a model's ability to perform multi-round multimodal referring and grounding, requiring logical consistency across dialogue turns while generating accurate text responses and bounding box coordinates for visual instances.

Datasets

• CB-LC — total ?; splits: test (-1)
• RefCOCOg — total ?; splits: test (-1)
• COCO 2017 — total ?; splits: test (-1)

Metrics

• BERT(·) (primary) — range: [0, 1]
  • BERTScore metric computing token-level cosine similarity between predicted and ground-truth text embeddings using a pre-trained BERT model.
• mIoU — range: [0, 1]
  • Mean Intersection over Union between predicted and ground-truth bounding boxes across all instances.
• Succ. Rate — range: [0, 1]
  • Fraction of test instances where the predicted bounding box IoU with the ground truth is at least 0.5.
• T — range: [0, 1]
  • Aggregate MRG score combining linguistic (BERT) and visual (IoU) consistency across all dialogue rounds.

Input / output format

Input: An image paired with a multi-turn dialogue history containing referring expressions (which may use pronouns like 'it' or 'the object' in later turns) and a current query.

Output: A natural language response and a bounding box coordinate set (or segmentation mask) localizing the referred object in the image.

Scoring recipe

def score(predictions, golds):
    bert_scores = [bertscore(p.text, g.text) for p, g in zip(predictions, golds)]
    ious = [box_iou(p.box, g.box) for p, g in zip(predictions, golds)]
    succ_rate = sum(1 for iou in ious if iou >= 0.5) / len(ious)
    mIoU = sum(ious) / len(ious)
    # T aggregates round-wise BERT and IoU scores across the dialogue thread
    T = compute_thread_aggregate(bert_scores, ious)
    return {'BERT': np.mean(bert_scores), 'mIoU': mIoU, 'Succ_Rate': succ_rate, 'T': T}

Common pitfalls

• Models are highly sensitive to prompt phrasing for visual grounding; performance varies significantly across variants like 'Where is the [name]?' vs 'Can you find the [name]?'.
• Multi-round evaluation requires resolving pronouns that depend on previous turns; failing to maintain dialogue context degrades performance to single-round levels.
• Direct comparison with LISA is complicated by its instability with explicit grounding tokens, which can produce segmentation masks with outliers that artificially lower box-level IoU.

Evidence (verbatim from paper)

We curate all threads in the test set of CB-LC into three question-and-answer pairs, where each round (except for the first one) is logically related to the previous rounds, thereby the difficulty increases round by round. In terms of the linguistic output, ChatterBox produces better BERT(·) scores than GPT4RoI, Kosmos-2, and LISA, and the advantage becomes more significant in the latter two rounds, implying its stronger ability in dealing with multi-round dialogues. Regarding the visual output... ChatterBox achieves the best IoU(·,·) scores throughout the entire thread... Combining the high quality of linguistic and visual output yields the better MRG scores (i.e., tn and T).

Citation

@misc{tian2024chatterbox,
  title={ChatterBox: Multi-round Multimodal Referring and Grounding},
  author={Tian et al. (2024)},
  year={2024},
  note={arXiv:2401.13307}
}

• arXiv: 2401.13307