Mteb Loco Jina Eval

mteb-loco-jina-eval

Nomic Embed: Training a Reproducible Long Context Text Embedder — Nussbaum et al. (2024) (arXiv:2402.01613, 2024)

What this evaluates

Evaluates text embedding models on short and long-context retrieval, clustering, and semantic similarity tasks to measure representation quality across varying sequence lengths.

Datasets

• MTEB — total ?; splits: test (-1)
• Jina Long Context Benchmark — total ?; splits: test (-1)
• LoCo Benchmark — total ?; splits: test (-1)

Metrics

• NDCG@10 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10, measuring retrieval effectiveness by weighting relevant documents by their graded relevance position.
• V-score — range: [0, 1]
  • Clustering performance metric reported for the Jina Long Context Benchmark, typically based on adjusted mutual information or similar clustering quality measures.
• Spearman's Tau — range: [-1, 1]
  • Rank correlation coefficient used to evaluate long-context retrieval tasks (Tau Scr., Tau Gov., Tau QMS.) and overall performance averages.

Input / output format

Input: Text strings (queries and documents) or single texts for clustering/STS tasks, with task-specific prefixes added (e.g., 'search_query', 'clustering').

Output: L2-normalized embedding vectors (except for Classification tasks where raw embeddings are used without normalization).

Scoring recipe

def compute_metrics(embeddings, labels, task_type):
    if task_type == 'retrieval':
        scores = cosine_similarity(query_emb, doc_emb)
        return ndcg_at_k(scores, labels, k=10)
    elif task_type == 'clustering':
        return v_score(cluster_labels, true_labels)
    elif task_type == 'long_context':
        return spearman_tau(predicted_ranks, true_ranks)
    elif task_type == 'classification':
        return accuracy(predicted_classes, true_labels)
    return 0

Common pitfalls

• Truncating all texts to 512 tokens for MTEB evaluation regardless of model context length.
• Adding task-specific prefixes (e.g., 'search_query', 'clustering') to inputs, which varies by task type.
• Skipping L2 normalization only for Classification tasks, while applying it to all other tasks.
• Using Dynamic NTK Interpolation (alpha=2) for sequences exceeding the 2048 training length.

Evidence (verbatim from paper)

Similar to Günther et al. ([2024]), we report the V-scores and NDCG@10 for the clustering and retrieval datasets respectively. We evaluate all models at sequence length 128, 512, and 8191.

Citation

@misc{nussbaum2024nomicembed,
  title={Nomic Embed: Training a Reproducible Long Context Text Embedder},
  author={Nussbaum et al. (2024)},
  year={2024},
  note={arXiv:2402.01613}
}

• arXiv: 2402.01613