Hifi Kpi Eval

hifi-kpi-eval

HiFi-KPI: A Dataset for Hierarchical KPI Extraction from Earnings Filings — Aavang et al. (2025) (arXiv:2502.15411, 2025)

What this evaluates

Evaluates models on hierarchical key performance indicator (KPI) extraction from SEC earnings filings, testing their ability to classify paragraph-level labels, perform token-level sequence labeling, and extract structured financial entities (tags, dates, currency, values) at varying granularities.

Datasets

• HiFi-KPI — total ?; splits: (unstated); repo https://github.com/aaunlp/HiFi-KPI
• HiFi-KPI Lite — total ?; splits: (unstated); repo https://github.com/aaunlp/HiFi-KPI

Metrics

• aggregated macro F1 (primary) — range: percent
  • Macro-averaged F1 score computed over all tags, aggregated across cumulative support (total count of included ground truth tags) to measure encapsulation at a given granularity.
• micro F1 — range: percent
  • Micro-averaged F1 score computed per entity label (e.g., Start Date, Currency, Value) by aggregating true positives, false positives, and false negatives across all instances before calculating precision and recall.
• Exact Match (EM) — range: percent
  • Percentage of instances where all predicted entity labels exactly match the gold labels for that instance.

Input / output format

Input: Paragraphs from SEC earnings filings (iXBRL format), optionally accompanied by hierarchical taxonomy context for granularity selection.

Output: For text classification: a single paragraph-level label. For sequence labeling: token-level entity tags. For LLM extraction: structured JSON containing tags, dates, currency, and numeric values.

Scoring recipe

def compute_metrics(predictions, golds):
    tp, fp, fn = 0, 0, 0
    for pred, gold in zip(predictions, golds):
        for p, g in zip(pred, gold):
            if p == g:
                tp += 1
            elif p != g:
                fp += 1
                fn += 1
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    em = sum(1 for p, g in zip(predictions, golds) if p == g) / len(predictions)
    return f1, em

Common pitfalls

• Cumulative support is defined as the total count of included ground truth tags, meaning macro F1 aggregation changes depending on the granularity level evaluated.
• Out-of-scope (OOS) tags are mapped to a single label during sequence labeling, which can artificially inflate precision/recall if the evaluation script does not treat OOS consistently with the training setup.
• Exact Match (EM) requires all entity types (tags, dates, currency, values) to be perfectly predicted per instance, making it significantly stricter than per-label micro F1.

Evidence (verbatim from paper)

We report the aggregated macro F1 over the cumulative support. Cumulative support defined as the total count of included ground truth tags. This is to showcase how good the model is at encapsulating a given amount of the dataset, at a given granularity.

Citation

@misc{aavang2025hifikpi,
  title={HiFi-KPI: A Dataset for Hierarchical KPI Extraction from Earnings Filings},
  author={Aavang et al. (2025)},
  year={2025},
  note={arXiv:2502.15411}
}

• arXiv: 2502.15411