Financial Phrase Bank Sentiment Eval

financial-phrase-bank-sentiment-eval

Can AI Read Between The Lines? Benchmarking LLMs On Financial Nuance — Kubica et al. (2025) (arXiv:2505.16090, 2025)

What this evaluates

Probes the ability of LLMs and traditional NLP tools to accurately classify financial sentiment (positive, neutral, or negative) from news headlines and earnings-related text. It specifically evaluates how well models capture nuanced, hedged, or domain-specific financial language compared to baseline sentiment engines.

Datasets

• Financial Phrase Bank — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correct sentiment classifications (positive, neutral, or negative) against the pre-labeled gold standard across all evaluated sentences.

Input / output format

Input: Financial sentences or headlines from the dataset, preprocessed for traditional NLP libraries, or fed via identical prompts to LLM-based tools.

Output: A single sentiment label: positive, neutral, or negative.

Scoring recipe

correct = 0
total = len(predictions)
for pred, gold in zip(predictions, gold_labels):
    if pred.strip().lower() in ['positive', 'neutral', 'negative'] and pred.strip().lower() == gold.strip().lower():
        correct += 1
accuracy = (correct / total) * 100

Common pitfalls

• LLM outputs often contain conversational filler or formatting; strict label extraction is required to avoid false negatives.
• Traditional NLP tools (e.g., TextBlob) default to neutral sentiment and miss domain-specific cues, requiring careful configuration or prompt engineering.
• Structured data (CSVs) fed to LLMs can cause hallucinations or formatting errors; converting to plain text is necessary for reliable evaluation.

Evidence (verbatim from paper)

After each model returned the sentiment of each sentence, accuracy was measured as the percentage of correct classifications against the pre-labeled dataset.

Citation

@misc{kubica2025financialnuance,
  title={Can AI Read Between The Lines? Benchmarking LLMs On Financial Nuance},
  author={Kubica et al. (2025)},
  year={2025},
  note={arXiv:2505.16090}
}

• arXiv: 2505.16090