# Financial Phrase Bank Sentiment Eval

> Probes the ability of LLMs and traditional NLP tools to accurately classify financial sentiment (positive, neutral, or negative) from news headlines and earnings-related text. It specifically evaluates how well models capture nuanced, hedged, or domain-specific financial language compared to baseline sentiment engines. Use when the user wants to benchmark on Financial Phrase Bank, or asks about evaluating this task. Reports accuracy.

- Skill: `qhjqhj00/financial-phrase-bank-sentiment-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/financial-phrase-bank-sentiment-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/financial-phrase-bank-sentiment-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/financial-phrase-bank-sentiment-eval

---


# financial-phrase-bank-sentiment-eval

> Can AI Read Between The Lines? Benchmarking LLMs On Financial Nuance — Kubica et al. (2025) (arXiv:2505.16090, 2025)

## What this evaluates

Probes the ability of LLMs and traditional NLP tools to accurately classify financial sentiment (positive, neutral, or negative) from news headlines and earnings-related text. It specifically evaluates how well models capture nuanced, hedged, or domain-specific financial language compared to baseline sentiment engines.

## Datasets

- **Financial Phrase Bank** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correct sentiment classifications (positive, neutral, or negative) against the pre-labeled gold standard across all evaluated sentences.

## Input / output format

**Input**: Financial sentences or headlines from the dataset, preprocessed for traditional NLP libraries, or fed via identical prompts to LLM-based tools.

**Output**: A single sentiment label: positive, neutral, or negative.

## Scoring recipe

```python
correct = 0
total = len(predictions)
for pred, gold in zip(predictions, gold_labels):
    if pred.strip().lower() in ['positive', 'neutral', 'negative'] and pred.strip().lower() == gold.strip().lower():
        correct += 1
accuracy = (correct / total) * 100
```

## Common pitfalls

- LLM outputs often contain conversational filler or formatting; strict label extraction is required to avoid false negatives.
- Traditional NLP tools (e.g., TextBlob) default to neutral sentiment and miss domain-specific cues, requiring careful configuration or prompt engineering.
- Structured data (CSVs) fed to LLMs can cause hallucinations or formatting errors; converting to plain text is necessary for reliable evaluation.

## Evidence (verbatim from paper)

> After each model returned the sentiment of each sentence, accuracy was measured as the percentage of correct classifications against the pre-labeled dataset.

## Citation

```bibtex
@misc{kubica2025financialnuance,
  title={Can AI Read Between The Lines? Benchmarking LLMs On Financial Nuance},
  author={Kubica et al. (2025)},
  year={2025},
  note={arXiv:2505.16090}
}
```

- arXiv: 2505.16090