Enterprise Benchmarks Eval

enterprise-benchmarks-eval

Enterprise Benchmarks for Large Language Model Evaluation — Bing Zhang et al. (2024) (arXiv:2410.12857, 2024)

What this evaluates

Evaluates LLMs on domain-specific enterprise tasks across finance, legal, climate, and cybersecurity. It probes capabilities like numerical reasoning, named entity recognition, document relevance ranking, and long-document summarization using real-world industry data.

Datasets

• Earnings Call Transcripts — total ?; splits: test (-1); repo https://github.com/IBM/helm-enterprise-benchmark
• News Headline — total ?; splits: test (-1); repo https://github.com/IBM/helm-enterprise-benchmark
• Credit Risk Assessment (NER) — total ?; splits: test (-1); repo https://github.com/IBM/helm-enterprise-benchmark
• KPI-Edgar — total ?; splits: test (-1); repo https://github.com/IBM/helm-enterprise-benchmark
• FiNER-139 — total ?; splits: test (-1); repo https://github.com/IBM/helm-enterprise-benchmark
• Opinion-based QA (FiQA) — total ?; splits: test (-1); repo https://github.com/IBM/helm-enterprise-benchmark
• Sentiment Analysis (FiQA SA) — total ?; splits: test (-1); repo https://github.com/IBM/helm-enterprise-benchmark
• Insurance QA — total ?; splits: test (-1); repo https://github.com/IBM/helm-enterprise-benchmark
• ConvFinQA — total ?; splits: test (-1); repo https://github.com/IBM/helm-enterprise-benchmark
• Financial Text Summarization (EDT) — total 303893; splits: test (-1); repo https://github.com/IBM/helm-enterprise-benchmark

Metrics

• Weighted F1 (primary) — range: [0, 1]
  • Weighted average of precision and recall across classes, accounting for class imbalance.
• Entity F1 — range: [0, 1]
  • F1 score computed at the entity level, matching predicted spans and labels to gold standard annotations.
• Adj F1 — range: [0, 1]
  • Adjusted F1 score for joint NER and relation extraction tasks, penalizing incorrect attribute linking.
• RR@10 — range: [0, 1]
  • Reciprocal Rank at top 10; 1/k where k is the position of the first relevant document in the ranked list.
• Accuracy — range: [0, 1]
  • Proportion of correctly predicted exact numerical values or class labels out of total instances.
• Rouge-L — range: [0, 1]
  • Longest Common Subsequence (LCS) based F1 score measuring overlap between generated summary and reference.

Input / output format

Input: Domain-specific text, documents, or tables provided with N-shot prompts (1, 5, 10, or 20 examples) tailored to the task type (classification, NER, QA, ranking, or summarization).

Output: Task-specific predictions: class labels, entity spans with tags, ranked document lists, exact numerical values, or abstractive summaries.

Scoring recipe

def compute_metric(predictions, gold, metric_name):
    if metric_name in ['Weighted F1', 'Entity F1', 'Adj F1']:
        return f1_score(gold, predictions, average='weighted')
    elif metric_name == 'RR@10':
        ranks = [i for i, p in enumerate(predictions[:10]) if p in gold]
        return 1.0 / (ranks[0] + 1) if ranks else 0.0
    elif metric_name == 'Accuracy':
        return sum(p == g for p, g in zip(predictions, gold)) / len(gold)
    elif metric_name == 'Rouge-L':
        return rouge_l_score(predictions, gold)
    return 0.0

Common pitfalls

• Using generic zero-shot prompts instead of the specified N-shot templates (1, 5, 10, or 20-shot) which are critical for domain adaptation.
• Confusing numerical NER with standard text NER, as financial/XBRL tags depend heavily on context rather than token surface form.
• Ignoring the strict dataset curation rule: benchmarks exclude any dataset without reference answers or with fewer than 100 test cases.

Evidence (verbatim from paper)

| Exact Value Match | Chain of Numeric Reasoning (ConvFinQA)Chen et al. ([2022]) | Multi-turn conversational finance question answering data for exploring the chain of numerical reasoning | 1-shot | Accuracy |

Citation

@misc{zhang2024enterprisebenchmarks,
  title={Enterprise Benchmarks for Large Language Model Evaluation},
  author={Bing Zhang et al. (2024)},
  year={2024},
  note={arXiv:2410.12857}
}

• arXiv: 2410.12857