Financial Retrieval Eval

financial-retrieval-eval

Greenback Bears and Fiscal Hawks: Finance is a Jungle and Text Embeddings Must Adapt — Anderson et al. (2024) (arXiv:2411.07142, 2024)

What this evaluates

Evaluates the ability of text embedding models to retrieve relevant financial document passages given complex, long-form queries. It probes domain-specific retrieval capabilities, including sensitivity to company names, tickers, financial metrics, and date-specific information.

Datasets

• Financial Document Retrieval Dataset — total 15200000; splits: train (14300000), val (444000), test (447000)

Metrics

• Recall@1 (primary) — range: [0, 1]
  • Recall@1 measures the fraction of queries for which the ground truth passage is ranked first among the retrieved candidates. Computed as (number of queries with GT passage in top-1) / (total number of queries).

Input / output format

Input: A natural language query (often long-form and complex) paired with a corpus of candidate text passages (max 512 tokens) augmented with a context line containing company name, ticker, and event/date.

Output: A ranked list of retrieved passages, with the primary metric computed on the top-1 result.

Scoring recipe

def compute_recall_at_1(predictions, gold):
    # predictions: list of top-1 passage IDs/text for each query
    # gold: list of ground truth passage IDs/text for each query
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)

Common pitfalls

• Data contamination can occur if passages from the same source document are split across train/val/test; the protocol enforces document-level splitting to prevent this.
• Synthetic queries generated by LLMs may differ in distribution and complexity from real user queries, potentially inflating retrieval performance on held-out synthetic data.
• Passages lack crucial context (company name, ticker, date) unless explicitly prepended, which is required for both finetuning and evaluation.

Evidence (verbatim from paper)

Passage retrieval results: Recall@1 on a held-out test split of 447K query-passage pairs. BAM embeddings finetuned for financial document retrieval significantly outperform general-purpose embeddings.

Citation

@misc{anderson2024greenback,
  title={Greenback Bears and Fiscal Hawks: Finance is a Jungle and Text Embeddings Must Adapt},
  author={Anderson et al. (2024)},
  year={2024},
  note={arXiv:2411.07142}
}

• arXiv: 2411.07142