Detailed Localized Captioning Eval

detailed-localized-captioning-eval

Describe Anything: Detailed Localized Image and Video Captioning — Lian et al. (2025) (arXiv:2504.16072, 2025)

What this evaluates

Evaluates a model's ability to generate detailed, region-specific descriptions for images and videos, ranging from keywords to multi-sentence captions. It probes fine-grained visual grounding, attribute recognition, and hallucination resistance by comparing generated text against reference captions or using attribute-level positive/negative judgments.

Datasets

• DLC-Bench — total ?; splits: test (-1)
• LVIS — total ?; splits: test (-1)
• PACO — total ?; splits: test (-1)
• Flickr30k Entities — total ?; splits: test (-1)
• Ref-L4 — total ?; splits: test (-1)
• HC-STVG — total ?; splits: test (-1)
• VideoRefer-Bench-D — total ?; splits: test (-1)

Metrics

• positive accuracy (primary) — range: [0, 1]
  • Fraction of test instances where the generated description correctly includes all specified positive attributes for the target region.
• negative accuracy — range: [0, 1]
  • Fraction of test instances where the generated description correctly excludes all specified negative (forbidden) attributes.
• average accuracy — range: [0, 1]
  • Arithmetic mean of positive accuracy and negative accuracy.
• semantic IoU — range: percent
  • Intersection over union of predicted and ground-truth semantic region masks.
• semantic similarity — range: percent
  • Cosine similarity between predicted and ground-truth semantic embeddings of the region.
• BLEU@4 — range: other
  • N-gram precision up to 4-grams with brevity penalty, standard for machine translation and captioning.
• METEOR — range: other
  • Harmonic mean of unigram precision and recall, incorporating stemming and synonymy matching.
• ROUGE-L — range: other
  • F-measure based on the longest common subsequence between prediction and reference.
• CIDEr — range: other
  • TF-IDF weighted n-gram similarity that downweights common words and rewards consensus among references.
• SPICE — range: other
  • Semantic Proposition Image Caption Evaluation score based on scene graph matching.

Input / output format

Input: Image or video frame(s) with a specified region (bounding box, mask, or focal crop) and a text prompt requesting a description at a certain granularity (keyword, phrase, or detailed).

Output: Text description of the specified region, ranging from a single keyword/phrase to multiple sentences.

Scoring recipe

def score_dlc_bench(predictions, gold):
    pos_correct = sum(1 for p, g in zip(predictions, gold) if matches_attributes(p, g.positive))
    neg_correct = sum(1 for p, g in zip(predictions, gold) if not contains_forbidden(p, g.negative))
    pos_acc = pos_correct / len(predictions)
    neg_acc = neg_correct / len(predictions)
    return {'positive_accuracy': pos_acc, 'negative_accuracy': neg_acc, 'average_accuracy': (pos_acc + neg_acc) / 2}

def score_captioning(predictions, gold):
    return {
        'BLEU@4': compute_bleu(predictions, gold, n=4),
        'METEOR': compute_meteor(predictions, gold),
        'ROUGE-L': compute_rouge(predictions, gold, 'rougeL'),
        'CIDEr': compute_cider(predictions, gold),
        'SPICE': compute_spice(predictions, gold)
    }

Common pitfalls

• Reference captions often omit valid details, causing models to be unfairly penalized on hallucination detection sub-tasks when they generate correct but unmentioned information.
• Using only local crops without cross-attention to global context severely degrades region-specific accuracy by losing scene semantics.
• Mixing prompt granularities (keyword vs. detailed) during evaluation inflates or deflates scores; prompts must strictly match the benchmark's required output length.

Evidence (verbatim from paper)

In the PACO benchmark, a challenging benchmark that includes both full objects and parts in complex scenes and requires the model to decide whether the region is an object or a part, our method achieves 73.2% semantic IoU and 84.2% semantic similarity, outperforming the previous best by 23.2% and 8.5% respectively.

Citation

@misc{lian2025describeanything,
  title={Describe Anything: Detailed Localized Image and Video Captioning},
  author={Lian et al. (2025)},
  year={2025},
  note={arXiv:2504.16072}
}

• arXiv: 2504.16072