Dreamlip Zero Shot Eval

dreamlip-zero-shot-eval

DreamLIP: Language-Image Pre-training with Long Captions — Kecheng Zheng et al. (arXiv:2403.17007, 2024)

What this evaluates

Evaluates the zero-shot transfer capability of language-image pre-trained models across image-text retrieval, semantic segmentation, image classification, and vision-language reasoning tasks.

Datasets

• ImageNet — total ?; splits: test (-1)
• MSCOCO — total ?; splits: test (-1)
• Flickr30K — total ?; splits: test (-1)
• ADE20K-150 — total ?; splits: test (-1)
• VOC-20 — total ?; splits: test (-1)

Metrics

• R@K (primary) — range: percent
  • Recall at top-K retrieved items. Computed by ranking texts/images by similarity score and checking if the ground truth appears in the top-K.
• Top-1 accuracy (primary) — range: percent
  • Percentage of images correctly classified by selecting the class label with the highest cosine similarity to the image embedding.
• mIOU — range: percent
  • Mean Intersection over Union across all semantic classes, measuring segmentation mask overlap with ground truth.

Input / output format

Input: Image resized to 224x224. Text truncated or padded to 77 tokens. For zero-shot classification, text inputs are predefined prompts containing class label names.

Output: Class label prediction (classification), ranked list of retrieved items (retrieval), pixel-wise class labels (segmentation).

Scoring recipe

def evaluate_zero_shot(image, text_prompts, gold_labels):
    img_emb = model.encode_image(image)
    txt_embs = model.encode_text(text_prompts)
    sim = (img_emb @ txt_embs.T) / temperature
    pred = text_prompts[torch.argmax(sim, dim=1)]
    return (pred == gold_labels).float().mean()

def evaluate_retrieval(img_embs, txt_embs, gold_indices, k):
    sim_matrix = img_embs @ txt_embs.T
    ranks = torch.argsort(sim_matrix, descending=True)
    recall = (ranks[:, :k].argmax(dim=1) == gold_indices).float().mean()
    return recall

Common pitfalls

• Using custom prompt templates instead of CLIP's predefined class-label prompts changes zero-shot results significantly.
• Text inputs must be strictly truncated/padded to 77 tokens to match the pre-trained text encoder's capacity.
• Evaluating semantic segmentation requires fine-tuning on COCO-stuff, not just zero-shot inference.

Evidence (verbatim from paper)

Following CLIP, we select 11 visual recognition datasets under the zero-shot setting... The same zero-shot classification protocol is applied following [47], which uses predefined prompts as text inputs. We use R@K to report the recall of top-K retrieval items. Top-1 accuracy is used for evaluation.

Citation

@misc{zheng2024dreamlip,
  title={DreamLIP: Language-Image Pre-training with Long Captions},
  author={Kecheng Zheng et al.},
  year={2024},
  note={arXiv:2403.17007}
}

• arXiv: 2403.17007