Hume Eval

hume-eval

HUME: Measuring the Human-Model Performance Gap in Text Embedding Tasks — El Assadi et al. (2025) (arXiv:2510.10062, 2025)

What this evaluates

Evaluates text embedding models against human baselines across 16 MTEB datasets, probing semantic similarity, classification, clustering, and reranking capabilities. It specifically measures cross-lingual performance and identifies task ambiguities where model scores may reflect label pattern reproduction rather than genuine understanding.

Datasets

• MTEB (16 datasets, 26 task-language pairs) — total ?; splits: test (-1); repo https://github.com/embeddings-benchmark/mteb

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly classified instances out of the total number of instances.
• Spearman correlation — range: [-1, 1]
  • Rank correlation coefficient between predicted and gold similarity scores.
• V-measure — range: [0, 1]
  • Clustering metric measuring homogeneity and completeness of cluster assignments.
• MAP — range: [0, 1]
  • Mean Average Precision for evaluating document reranking quality.

Input / output format

Input: Text pairs (query, document) for STS and reranking tasks; single text instances for classification and clustering tasks.

Output: Model-generated embeddings or predicted labels/rankings/similarity scores per task.

Scoring recipe

def compute_metrics(predictions, gold, task_type):
    if task_type == 'classification':
        return accuracy_score(gold, predictions) * 100
    elif task_type == 'clustering':
        return v_measure_score(gold, predictions) * 100
    elif task_type == 'reranking':
        return map_score(gold, predictions) * 100
    elif task_type == 'sts':
        return spearmanr(gold, predictions)[0] * 100
    return mean([compute_metrics(p, g, t) for p, g, t in zip(preds, golds, tasks)])

Common pitfalls

• Task ambiguity in emotion classification and academic clustering leads to low human agreement, making 'superhuman' model scores potentially reflect pattern matching rather than true understanding.
• Cross-lingual performance heavily depends on training data dominance; English tasks favor models while non-English tasks (Arabic, Russian) favor humans due to cultural/linguistic knowledge.
• Aggregating heterogeneous metrics (accuracy, V-measure, MAP, Spearman) into a single score requires careful normalization, which may obscure task-specific strengths and weaknesses.

Evidence (verbatim from paper)

Human annotators rank 4th overall with a score of 0.776, trailing 3 large models but outperforming 10 others. This positioning reveals that humans neither represent a performance ceiling nor a lower bound, but rather occupy a middle ground that varies significantly by task category and language. Classification: Human performance averages 70.3 across classification tasks, with substantial variation reflecting task-specific challenges. Clustering presents the greatest challenge for humans, averaging 67.4 V-measure with extreme variation. Reranking: Humans achieve strong performance in reranking (87.2 average MAP), demonstrating intuitive document relevance understanding. STS: Human performance averages 83.2 Spearman correlation, with notable dataset-specific variation.

Citation

@misc{elassadi2025hume,
  title={HUME: Measuring the Human-Model Performance Gap in Text Embedding Tasks},
  author={El Assadi et al. (2025)},
  year={2025},
  note={arXiv:2510.10062}
}

• arXiv: 2510.10062