Clip Mteb Eval

clip-mteb-eval

Jina CLIP: Your CLIP Model Is Also Your Text Retriever — Koukounas et al. (2024) (arXiv:2405.20204, 2024)

What this evaluates

Evaluates cross-modal (text-image) retrieval and text-only embedding performance. Probes zero-shot retrieval accuracy, semantic similarity, and overall text embedding capability across diverse benchmarks.

Datasets

• CLIP Benchmark — total ?; splits: test (-1)
• MTEB — total ?; splits: test (-1)

Metrics

• r@5 (primary) — range: percent
  • Recall@5: fraction of queries where the ground-truth relevant item appears in the top-5 retrieved results. Averaged across all queries.
• ndcg@10 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10. Measures ranking quality by weighting relevant items higher at top positions.
• spearman — range: [-1, 1]
  • Spearman rank correlation coefficient between predicted similarity scores and human-annotated relevance scores.
• Avg MTEB Score — range: percent
  • Macro-average of task-specific scores (normalized to 0-100) across the 8 MTEB tasks.

Input / output format

Input: Text queries paired with image captions/descriptions for cross-modal retrieval; text queries and documents for text-only retrieval; text pairs for semantic textual similarity (STS).

Output: Ranked list of retrieved items (images or documents) or similarity scores for text pairs.

Scoring recipe

def recall_at_k(retrieved, relevant, k=5):
    if not relevant: return 0.0
    hits = sum(1 for item in retrieved[:k] if item in relevant)
    return hits / len(relevant)

def ndcg_at_k(retrieved, relevant, k=10):
    dcg = sum(1.0 / math.log2(i + 2) for i, item in enumerate(retrieved[:k]) if item in relevant)
    ideal = sorted([1.0 / math.log2(i + 2) for i in range(min(len(relevant), k))], reverse=True)
    idcg = sum(ideal)
    return dcg / idcg if idcg > 0 else 0.0

Common pitfalls

• Recall@5 is computed per query and then averaged across the dataset, not calculated globally over all retrieved items.
• MTEB scores are task-specific averages normalized to [0,100] before computing the overall MTEB average; raw correlation values must be transformed first.
• Cross-modal retrieval is evaluated in a zero-shot setting without fine-tuning on the benchmark datasets.

Evidence (verbatim from paper)

txt-img r@5 : Text to Image Recall@5 [%] img-txt r@5 : Image to Text Recall@5 [%] r@5 : Recall@5 [%]
spearman: Spearman Correlation

For zero-shot image-text and text-image information retrieval, we evaluate using Flickr8k, Flickr30K and MSCOCO Captions, which are all included in CLIP Benchmark. [jina-clip-v1] achieves an average Recall@5 of 85.8% across all retrieval benchmarks, outperforming OpenAI’s CLIP model and performing on par with EVA-CLIP.

Citation

@misc{koukounas2024jinaclip,
  title={Jina CLIP: Your CLIP Model Is Also Your Text Retriever},
  author={Koukounas et al. (2024)},
  year={2024},
  note={arXiv:2405.20204}
}

• arXiv: 2405.20204