Finmteb Eval

finmteb-eval

Do We Need Domain-Specific Embedding Models? An Empirical Investigation — Tang et al. (2024) (arXiv:2409.18511, 2024)

What this evaluates

Evaluates embedding models on a finance-specific benchmark across seven standard embedding tasks to measure performance relative to general-domain benchmarks. It probes whether general-purpose models suffer a significant performance drop on domain-specific financial text and investigates if this gap is driven by domain shift or inherent dataset complexity.

Datasets

• FinMTEB — total ?; splits: (unstated)

Metrics

• FinMTEB Score (primary) — range: [0, 1] or percent
  • The arithmetic mean of task-specific evaluation scores across all 64 datasets in the benchmark. Each dataset is scored using the standard MTEB metric appropriate for its task (e.g., Spearman correlation for STS, accuracy/F1 for classification, NDCG@10 for retrieval).

Input / output format

Input: Varies by task: sentence pairs for STS and Pair-Classification, query-document pairs for Retrieval and Reranking, labeled documents for Classification, unlabeled document sets for Clustering, and document-summary pairs for Summarization.

Output: Task-specific predictions: similarity scores, ranked document lists, class labels, cluster assignments, or generated summaries, depending on the task.

Scoring recipe

scores = []
for dataset in finmteb_datasets:
    if dataset.task == "STS":
        scores.append(spearman_corr(predictions, gold))
    elif dataset.task == "Classification":
        scores.append(accuracy(predictions, gold))
    elif dataset.task == "Retrieval":
        scores.append(ndcg_at_k(predictions, gold, k=10))
    elif dataset.task == "Reranking":
        scores.append(mrr(predictions, gold))
    elif dataset.task == "Clustering":
        scores.append(v_measure(predictions, gold))
    elif dataset.task == "Pair-Classification":
        scores.append(accuracy(predictions, gold))
    elif dataset.task == "Summarization":
        scores.append(rouge_l(predictions, gold))
finmteb_score = sum(scores) / len(scores)

Common pitfalls

• Assuming the performance drop on FinMTEB is purely due to domain shift without first controlling for linguistic complexity and dataset difficulty.
• Averaging scores across tasks without using the standardized MTEB metric for each specific task type, which can skew results due to different score scales.
• Overgeneralizing ANOVA results: Pair-classification showed no significant model effect, unlike other tasks where model choice heavily influenced scores.

Evidence (verbatim from paper)

The main scoring metric for each task is the same as used with that of the MTEB benchmark, and the details are presented in the Appendix [E]. The “FinMTEB Score” column shows the average performance score evaluated on the proposed FinMTEB benchmark.

Citation

@misc{tang2024domain,
  title={Do We Need Domain-Specific Embedding Models? An Empirical Investigation},
  author={Tang et al. (2024)},
  year={2024},
  note={arXiv:2409.18511}
}

• arXiv: 2409.18511