Fintagging Eval

fintagging-eval

FinTagging: Benchmarking LLMs for Extracting and Structuring Financial Information — Wang et al. (2025) (arXiv:2505.20650, 2025)

What this evaluates

Evaluates large language models on structure-aware XBRL tagging for financial information. It probes two subtasks: numeric entity identification (FinNI) and fine-grained concept linking (FinCL) against the US-GAAP taxonomy, testing the model's ability to extract structured facts and align them with hierarchical financial concepts.

Datasets

• FinTagging — total ?; splits: test (-1); repo https://github.com/The-FinAI/FinTagging

Metrics

• macro-F1 (primary) — range: [0, 1]
  • Harmonic mean of macro-precision and macro-recall, computed as 2 * (P * R) / (P + R). Macro averaging treats all tags equally regardless of frequency.
• micro-F1 — range: [0, 1]
  • Harmonic mean of micro-precision and micro-recall, computed as 2 * (P * R) / (P + R). Micro averaging weights each instance equally, reflecting performance on frequent labels.
• Accuracy — range: [0, 1]
  • Proportion of correctly linked taxonomy concepts out of total instances.

Input / output format

Input: Financial text and tables provided via a prompt template. For the FinCL subtask, a candidate list of taxonomy concepts is retrieved beforehand.

Output: Structured triplet (Tag, Fact, Type) for FinNI, and a single selected US-GAAP taxonomy concept from the candidate list for FinCL.

Scoring recipe

def compute_f1(precision, recall):
    return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)

Common pitfalls

• Relying solely on precision for FinNI, as missing facts are more damaging than producing a few spurious ones.
• Treating XBRL tagging as single-step extreme classification, which causes all models to collapse to zero F1.
• Ignoring error propagation in the two-stage pipeline, where extraction mistakes cap downstream linking accuracy.

Evidence (verbatim from paper)

From a macro perspective, which emphasizes balanced performance across frequent and rare tags, large general-purpose LLMs clearly dominate. DeepSeek-V3, GPT-4o, and Llama-4-Scout achieve the strongest macro-F1 scores, surpassing all fine-tuned PLMs and indicating better generalization to long-tail concepts.

Citation

@misc{wang2025fintagging,
  title={FinTagging: Benchmarking LLMs for Extracting and Structuring Financial Information},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2505.20650}
}

• arXiv: 2505.20650