Docfinqa Eval

docfinqa-eval

DocFinQA: A Long-Context Financial Reasoning Dataset — Varshini Reddy et al. (2024) (arXiv:2401.06915, 2024)

What this evaluates

Evaluates long-context financial reasoning and context retrieval capabilities. It probes whether models can accurately retrieve relevant sections from lengthy SEC reports and answer numerical questions grounded in those documents.

Datasets

• DocFinQA — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• HR@k (primary) — range: [0, 1]
  • Hit Rate at k: the fraction of queries where the ground-truth relevant chunk is ranked within the top-k retrieved chunks.
• accuracy — range: percent
  • Exact match accuracy of the generated answer against the ground-truth numerical answer or Python program.

Input / output format

Input: For retrieval: a question q and a set of document chunks C. For QA: a question q and k retrieved chunks provided as in-context examples.

Output: For retrieval: the index or rank of the retrieved chunk. For QA: a natural language or numerical answer.

Scoring recipe

# Retrieval HR@k
hr_at_k = sum(1 for q in queries if target_chunk_idx in top_k_indices[q]) / len(queries)
# QA Accuracy
accuracy = sum(1 for q in queries if normalize(generated_answer[q]) == normalize(gold_answer[q])) / len(queries)

Common pitfalls

• Retrieval performance is highly sensitive to the embedding model; fine-tuned ColBERT significantly outperforms pre-trained baselines.
• LLM accuracy strongly depends on the few-shot configuration and context length, with generic LLMs preferring shorter contexts + more shots, while code-trained LLMs prefer longer contexts.
• Proprietary models like GPT-3 and BloombergGPT were excluded due to API inaccessibility, limiting direct comparison with the latest commercial baselines.

Evidence (verbatim from paper)

We evaluate these models using HR@k on the test set of DocFinQA using the target $c^{igstar}.

Citation

@misc{reddy2024docfinqa,
  title={DocFinQA: A Long-Context Financial Reasoning Dataset},
  author={Varshini Reddy et al. (2024)},
  year={2024},
  note={arXiv:2401.06915}
}

• arXiv: 2401.06915