Jina Embeddings V4 Eval

jina-embeddings-v4-eval

jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrieval — Günther et al. (2025) (arXiv:2506.18902, 2025)

What this evaluates

Evaluates a multimodal embedding model's ability to retrieve relevant documents, images, and code from large corpora, and to measure semantic similarity between text pairs across multiple languages and modalities.

Datasets

• J-VDR — total ?; splits: test (-1)
• ViDoRe — total ?; splits: test (-1)
• CLIPB — total ?; splits: test (-1); repo https://github.com/LAION-AI/CLIP_benchmark
• MMTEB — total ?; splits: test (-1)
• MTEB-en — total ?; splits: test (-1)
• COIR — total ?; splits: test (-1)
• LEMB — total ?; splits: test (-1)
• STS-m — total ?; splits: test (-1)
• STS-en — total ?; splits: test (-1)

Metrics

• nDCG@10 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at cutoff K=10. Computes the weighted sum of relevance scores for retrieved documents, normalized by the ideal DCG. Higher is better.
• nDCG@5 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at cutoff K=5. Same as nDCG@10 but truncated at the top 5 retrieved results.
• Spearman coefficient — range: [-1, 1]
  • Spearman rank correlation coefficient measuring the monotonic relationship between predicted similarity scores and ground-truth human judgments.

Input / output format

Input: Query and document pairs (text, images, or code) passed through the embedding model to generate vector representations.

Output: Ranked list of documents for retrieval tasks; cosine similarity scores for STS tasks.

Scoring recipe

def compute_ndcg_at_k(relevant_docs, retrieved_docs, k):
    dcg = sum(1 / math.log2(i + 2) for i, doc in enumerate(retrieved_docs[:k]) if doc in relevant_docs)
    idcg = sum(1 / math.log2(i + 2) for i in range(min(k, len(relevant_docs))))
    return dcg / idcg if idcg > 0 else 0.0

def compute_spearman(gold_scores, pred_scores):
    return scipy.stats.spearmanr(gold_scores, pred_scores).statistic

# For J-VDR/ViDoRE: average multilingual task scores first, then average across all tasks.

Common pitfalls

• Different benchmarks use different cutoffs: nDCG@5 for J-VDR, ViDoRe, and CLIPB, but nDCG@10 for MMTEB, MTEB-en, COIR, and LEMB.
• J-VDR and ViDoRE require a two-step averaging process: calculate the average for multilingual tasks first, then average that result across all tasks.
• Specific tasks like ArguAna require a fixed query prefix ('Given a claim, find documents that refute the claim') to match the official evaluation protocol.

Evidence (verbatim from paper)

Scores are nDCG@5 for J-VDR, ViDoRe, and CLIPB, and nDCG@10 for MMTEB, MTEB-en, COIR, and LEMB, and Spearman coefficient for STS-m and STS-en. Average Calculation: For J-VDR and ViDoRE, we calculate the average for the multilingual tasks first and consider this as a single score before calculating the average across all tasks.

Citation

@misc{gunther2025jinaembeddingsv4,
  title={jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrieval},
  author={Günther et al. (2025)},
  year={2025},
  note={arXiv:2506.18902}
}

• arXiv: 2506.18902