Finfre RAG Eval

finfre-rag-eval

Understanding Structured Financial Data with LLMs: A Case Study on Fraud Detection — Tan et al. (2025) (arXiv:2512.13040, 2025)

What this evaluates

Evaluates LLMs on tabular financial fraud detection by testing their ability to classify transactions using retrieval-augmented in-context learning and importance-guided feature reduction. It probes whether providing a compact set of high-impact features and relevant historical examples improves classification under severe class imbalance.

Datasets

• CCF — total ?; splits: test (8000), val (2000), retrieval_pool (-1)
• CCFRAUD — total ?; splits: test (8000), val (2000), retrieval_pool (-1)
• IEEE-CIS — total ?; splits: test (8000), val (2000), retrieval_pool (-1)
• PAYSIM — total ?; splits: test (8000), val (2000), retrieval_pool (-1)

Metrics

• F1-score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall).
• Matthews Correlation Coefficient (MCC) (primary) — range: [-1, 1]
  • Correlation coefficient between observed and predicted binary classifications, robust to class imbalance. Ranges from -1 to +1.
• Precision — range: [0, 1]
  • Share of flagged transactions that are truly fraudulent.
• Recall — range: [0, 1]
  • Share of frauds correctly identified.

Input / output format

Input: A single tabular financial transaction represented as a query instance with a subset of top-k features (default k=10), augmented with n=20 nearest-neighbor historical transactions from a retrieval pool as in-context examples.

Output: A 5-point risk score (1–5) indicating fraud probability, accompanied by a brief natural language explanation. For binary baselines, a direct 'bad' or 'good' classification.

Scoring recipe

pred_binary = 1 if risk_score >= 4 else 0
tp = sum(1 for p, g in zip(pred_binary, gold) if p == 1 and g == 1)
fp = sum(1 for p, g in zip(pred_binary, gold) if p == 1 and g == 0)
fn = sum(1 for p, g in zip(pred_binary, gold) if p == 0 and g == 1)
tn = sum(1 for p, g in zip(pred_binary, gold) if p == 0 and g == 0)
precision = tp / (tp + fp) if (tp + fp) > 0 else 0
recall = tp / (tp + fn) if (tp + fn) > 0 else 0
f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
mcc = (tp * tn - fp * fn) / ((tp+fp)*(tp+fn)*(tn+fp)*(tn+fn))**0.5

Common pitfalls

• Using accuracy as a metric, which is misleading due to severe class imbalance in fraud datasets.
• Prompting LLMs directly on raw, high-dimensional tabular data without feature reduction or retrieval, which yields near-random performance (negative MCC).
• Ignoring the fixed decision threshold: the paper explicitly treats any risk score ≥ 4 as a positive fraud prediction; changing this threshold drastically alters F1/MCC.

Evidence (verbatim from paper)

Due to the class imbalance, accuracy is not informative. We therefore use F1-score and Matthews Correlation Coefficient (MCC) (Chicco and Jurman, 2020) as primary metrics, as they better reflect performance under imbalance. We also report precision (share of flagged transactions that are truly fraudulent) and recall (share of frauds correctly identified) for analyst reference.

Citation

@misc{tan2025understanding,
  title={Understanding Structured Financial Data with LLMs: A Case Study on Fraud Detection},
  author={Tan et al. (2025)},
  year={2025},
  note={arXiv:2512.13040}
}

• arXiv: 2512.13040