Dsd Scene Analysis Eval

dsd-scene-analysis-eval

Peer-Ranked Precision: Creating a Foundational Dataset for Fine-Tuning Vision Models from DataSeeds' Annotated Imagery — Sajjad Abdoli et al. (2025) (arXiv:2506.05673, 2025)

What this evaluates

Evaluates the ability of vision-language models to generate detailed, technically accurate scene descriptions from images, leveraging high-fidelity human annotations and peer-ranked photography data.

Datasets

• DataSeeds.AI Sample Dataset (DSD) — total 10610; splits: train (9549), val (1061)

Metrics

• BLEU-4 (primary) — range: [0, 1]
  • Measures 4-gram precision between generated and reference texts, scaled by a brevity penalty (BP) that penalizes outputs shorter than the reference.
• ROUGE-L — range: [0, 1]
  • Computes the recall of the longest common subsequence (LCS) between generated and reference texts, preserving sequence order for fluency evaluation.
• BERTScore F1 — range: [0, 1]
  • Calculates the harmonic mean of token-level precision and recall using cosine similarity between contextual BERT embeddings of generated and reference texts.
• CLIPscore — range: percent
  • Computes 100 times the cosine similarity between L2-normalized image and text embeddings from a Long-CLIP model to measure semantic alignment.

Input / output format

Input: Image paired with a structured prompt requesting a scene analysis (20-80 words) covering context, environment, lighting, camera angle, color palette, photography style, and visible text.

Output: A textual scene description generated by the model, evaluated against human-annotated ground truth descriptions (15+ words narrative, 20-30 words technical analysis).

Scoring recipe

def compute_metrics(predictions, references, images=None):
    bleu4 = nltk.translate.bleu_score.sentence_bleu([ref.split()], pred.split(), weights=(0,0,0,1))
    rouge_l = rouge.rouge_l(references, predictions, metric_type='l')
    _, _, bert_f1 = bert_score.score([pred], [ref], lang='en')
    if images is not None:
        img_emb = longclip.encode_image(images)
        txt_emb = longclip.encode_text(texts)
        clip_score = 100 * cosine_similarity(img_emb, txt_emb)
    return {'BLEU-4': bleu4, 'ROUGE-L': rouge_l, 'BERTScore_F1': bert_f1, 'CLIPscore': clip_score}

Common pitfalls

• Checkpoint selection differs by model: LLaVA-NEXT uses minimum validation loss, while BLIP2 uses an aggregate metric (CIDEr + BLEU-4) peaking at epoch 1.
• Standard CLIP models truncate text at 77 tokens; this evaluation requires Long-CLIP to handle longer generated captions.
• AWS Rekognition baseline strictly uses a 50% confidence threshold for label detection, which may skew precision/recall comparisons.

Evidence (verbatim from paper)

We employed the following metrics to evaluate the generated descriptions from the fine-tuned model against human annotations. The BLEU (Papineni et al. [[2002]]) score measures the precision of n-gram matches between generated and reference texts, with particular emphasis on 4-gram matches (BLEU-4) to capture phrase-level accuracy. The ROUGE-L (Lin [[2004]]) metric evaluates the recall of the longest common subsequences, providing insight into the structural similarity between generated and reference descriptions. Additionally, we utilized BERTScore (Zhang et al. [[2019]]), which leverages contextual embeddings from pre-trained language models to compute semantic similarity beyond surface-level token matching.

Citation

@misc{abdoli2025peer,
  title={Peer-Ranked Precision: Creating a Foundational Dataset for Fine-Tuning Vision Models from DataSeeds' Annotated Imagery},
  author={Sajjad Abdoli et al. (2025)},
  year={2025},
  note={arXiv:2506.05673}
}

• arXiv: 2506.05673