Mteb Longembed Eval

mteb-longembed-eval

Linear-Time and Constant-Memory Text Embeddings Based on Recurrent Language Models — Grantner et al. (2026) (arXiv:2604.18199, 2026)

What this evaluates

Evaluates text embedding models on English, multilingual, and long-context retrieval tasks to measure semantic similarity, classification, clustering, reranking, and retrieval performance.

Datasets

• MTEB(eng, v2) — total ?; splits: test (-1)
• MTEB(Multilingual, v2) — total ?; splits: test (-1)
• LongEmbed — total ?; splits: test (-1)

Metrics

• mean over tasks (primary) — range: [0, 1] | percent
  • Arithmetic mean of task-specific evaluation scores (e.g., accuracy, F1, nDCG, Pearson/Spearman correlation for STS) across all tasks in the benchmark.

Input / output format

Input: Text pairs or single texts for retrieval, classification, clustering, reranking, STS, and summarization tasks, with sequence lengths up to 32,768 tokens for LongEmbed.

Output: Fixed-dimensional text embeddings (vectors) generated by the model.

Scoring recipe

def compute_mean_over_tasks(predictions, golds, task_metrics):
    scores = []
    for task in tasks:
        scores.append(task_metrics[task](predictions[task], golds[task]))
    return sum(scores) / len(scores)

Common pitfalls

• Aggregating diverse task metrics (classification, retrieval, STS) into a single mean score can obscure model weaknesses on specific capabilities.
• Long-context benchmarks like LongEmbed require careful handling of sequence length limits and memory constraints, which may affect fair comparison between recurrent and transformer architectures.
• Differences in base model pretraining scale and data composition can confound architectural comparisons between recurrent and transformer embedders.

Evidence (verbatim from paper)

The MTEB(eng, v2) benchmark includes 41 English datasets covering classification, clustering, pair classification, reranking, retrieval, semantic textual similarity (STS), and summarization tasks. The MTEB(Multilingual, v2) benchmark extends this to a multilingual setting encompassing more than 250 languages, and LongEmbed evaluates retrieval performance on long documents with sequences of up to 32,768 tokens in our setup. Results are aggregated as the mean over tasks and the mean over task types where applicable.

Citation

@misc{grantner2026lineartime,
  title={Linear-Time and Constant-Memory Text Embeddings Based on Recurrent Language Models},
  author={Grantner et al. (2026)},
  year={2026},
  note={arXiv:2604.18199}
}

• arXiv: 2604.18199