Financial Nlp Efficiency Eval

financial-nlp-efficiency-eval

Is GPT-OSS All You Need? Benchmarking Large Language Models for Financial Intelligence and the Surprising Efficiency Paradox — Bi et al. (2025) (arXiv:2512.14717, 2025)

What this evaluates

Evaluates large language models on ten financial NLP tasks to measure classification and generation accuracy, inference speed, and a novel Token Efficiency Score (TES) that quantifies the performance-compute trade-off.

Datasets

• Financial NLP tasks (10 datasets) — total ?; splits: test (-1)

Metrics

• Token Efficiency Score (TES) (primary) — range: other
  • TES = (1000 / tokens) × 100 if the model produces a correct answer, otherwise 0. It quantifies efficiency by inversely scaling with token consumption while rewarding correctness.
• Accuracy — range: percent
  • Percentage of correctly predicted labels or answers across the ten financial NLP tasks.

Input / output format

Input: Financial text samples for ten NLP tasks (e.g., sentiment analysis, question answering, entity recognition). Numerical values are preserved in their original format rather than converted to text.

Output: Deterministic/greedy decoded text responses, maximum 512 tokens, with early stopping on end-of-sequence tokens.

Scoring recipe

def compute_tes_and_accuracy(predictions, golds, tokens_used):
    correct = sum(1 for p, g in zip(predictions, golds) if p == g)
    accuracy = correct / len(golds)
    tes_values = []
    for p, g, t in zip(predictions, golds, tokens_used):
        if p == g:
            tes_values.append((1000 / t) * 100)
        else:
            tes_values.append(0)
    return accuracy, sum(tes_values) / len(tes_values)

Common pitfalls

• TES heavily penalizes longer correct answers due to the inverse token relationship, making it sensitive to verbosity rather than just correctness.
• Greedy decoding (temperature=0) is strictly enforced, so results from sampling-based decoding are not directly comparable.
• Zero-shot evaluation yields lower absolute accuracy than fine-tuned models in literature, so cross-study comparisons require caution.

Evidence (verbatim from paper)

We implement greedy decoding for all generation tasks to eliminate randomness and enable reproducibility. Maximum generation length is set to 512 tokens for question answering tasks, though most responses require fewer than 100 tokens. ... TES: Token Efficiency Score = (1000/tokens) × 100 if correct, 0 if the model cannot produce answers

Citation

@misc{bi2025gptoss,
  title={Is GPT-OSS All You Need? Benchmarking Large Language Models for Financial Intelligence and the Surprising Efficiency Paradox},
  author={Bi et al. (2025)},
  year={2025},
  note={arXiv:2512.14717}
}

• arXiv: 2512.14717