# Fintagging Eval

> fintagging-eval

- Skill: `qhjqhj00/fintagging-eval` (Agent Skill)
- Install (CLI): `npx skillmds@latest add qhjqhj00/fintagging-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/fintagging-eval/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-21
- Page: https://skillmd.com/skills/qhjqhj00/fintagging-eval

---


# fintagging-eval

> FinTagging: Benchmarking LLMs for Extracting and Structuring Financial Information — Wang et al. (2025) (arXiv:2505.20650, 2025)

## What this evaluates

Evaluates large language models on structure-aware XBRL tagging for financial information. It probes two subtasks: numeric entity identification (FinNI) and fine-grained concept linking (FinCL) against the US-GAAP taxonomy, testing the model's ability to extract structured facts and align them with hierarchical financial concepts.

## Datasets

- **FinTagging** — total ?; splits: test (-1); repo https://github.com/The-FinAI/FinTagging

## Metrics

- `macro-F1` **(primary)** — range: [0, 1]
  - Harmonic mean of macro-precision and macro-recall, computed as 2 * (P * R) / (P + R). Macro averaging treats all tags equally regardless of frequency.
- `micro-F1` — range: [0, 1]
  - Harmonic mean of micro-precision and micro-recall, computed as 2 * (P * R) / (P + R). Micro averaging weights each instance equally, reflecting performance on frequent labels.
- `Accuracy` — range: [0, 1]
  - Proportion of correctly linked taxonomy concepts out of total instances.

## Input / output format

**Input**: Financial text and tables provided via a prompt template. For the FinCL subtask, a candidate list of taxonomy concepts is retrieved beforehand.

**Output**: Structured triplet (Tag, Fact, Type) for FinNI, and a single selected US-GAAP taxonomy concept from the candidate list for FinCL.

## Scoring recipe

```python
def compute_f1(precision, recall):
    return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)
```

## Common pitfalls

- Relying solely on precision for FinNI, as missing facts are more damaging than producing a few spurious ones.
- Treating XBRL tagging as single-step extreme classification, which causes all models to collapse to zero F1.
- Ignoring error propagation in the two-stage pipeline, where extraction mistakes cap downstream linking accuracy.

## Evidence (verbatim from paper)

> From a macro perspective, which emphasizes balanced performance across frequent and rare tags, large general-purpose LLMs clearly dominate. DeepSeek-V3, GPT-4o, and Llama-4-Scout achieve the strongest macro-F1 scores, surpassing all fine-tuned PLMs and indicating better generalization to long-tail concepts.

## Citation

```bibtex
@misc{wang2025fintagging,
  title={FinTagging: Benchmarking LLMs for Extracting and Structuring Financial Information},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2505.20650}
}
```

- arXiv: 2505.20650