Verafi Financial QA Eval

verafi-financial-qa-eval

VERAFI: Verified Agentic Financial Intelligence through Neurosymbolic Policy Generation — Akinfaderin et al. (2025) (arXiv:2512.14744, 2025)

What this evaluates

Probes an agentic RAG system's ability to retrieve relevant SEC filings and generate factually correct, complete financial answers. It specifically tests the impact of neurosymbolic policy validation on suppressing hallucinations and mathematical errors in high-stakes financial domains.

Datasets

• FinanceBench-style Financial QA Dataset — total ?; splits: test (-1)

Metrics

• Factual Correctness (primary) — range: [0, 1]
  • Score assigned by an LLM-as-a-Judge (Claude 3.7 Sonnet v1 via Amazon Bedrock) evaluating whether generated responses accurately reflect retrieved financial information and answer the posed questions. Ranges from 0 to 1.
• Recall@3 — range: [0, 1]
  • Proportion of queries where at least one relevant financial passage is retrieved within the top 3 results.
• NDCG@3 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 3, measuring the quality of the ranked retrieval list with position-based discounting.
• Completeness — range: [0, 1]
  • Score from the LLM-as-a-Judge assessing whether the generated response fully addresses all aspects of the financial query.

Input / output format

Input: Financial queries paired with a vector database of SEC filings (10-K, 10-Q, 8-K) chunked into 500-character segments with 50-character overlap.

Output: Generated natural language financial responses/answers to the queries.

Scoring recipe

def evaluate_retrieval(retrieved_docs, relevant_docs, k=3):
    hit = 1 if any(d in relevant_docs for d in retrieved_docs[:k]) else 0
    recall = len(set(retrieved_docs[:k]) & set(relevant_docs)) / max(len(relevant_docs), 1)
    return recall, ndcg_at_k(retrieved_docs, relevant_docs, k), mrr_at_k(retrieved_docs, relevant_docs), hit

def evaluate_generation(prediction, query, context):
    judge_prompt = f'Judge factual correctness and completeness: Context={context}, Query={query}, Answer={prediction}'
    score = bedrock_llm_evaluate(judge_prompt)
    return score

Common pitfalls

• LLM-as-a-Judge scores are highly sensitive to the judge model and prompt design; results are not directly comparable to RAGAS/GPT-4 baselines without re-evaluation.
• Retrieval metrics are strictly capped at k=3, which may mask performance degradation on longer document lists or different cutoff thresholds.
• The evaluation dataset covers only four specific companies across limited years, potentially biasing results toward those specific reporting styles and industries.

Evidence (verbatim from paper)

We assess VERAFI performance using two complementary evaluation frameworks. For retrieval effectiveness, we employ standard information retrieval metrics including Recall@3, NDCG@3, MRR@3, and Hit Rate@3, measuring the system’s ability to identify relevant financial passages for complex queries. For generation quality assessment, we utilize LLM-as-a-Judge evaluation to measure factual correctness and completeness of generated financial responses.

Citation

@misc{akinfaderin2025verafi,
  title={VERAFI: Verified Agentic Financial Intelligence through Neurosymbolic Policy Generation},
  author={Akinfaderin et al. (2025)},
  year={2025},
  note={arXiv:2512.14744}
}

• arXiv: 2512.14744