# Financial LLM Zero Shot Eval

> financial-llm-zero-shot-eval

- Skill: `qhjqhj00/financial-llm-zero-shot-eval` (Agent Skill)
- Install (CLI): `npx skillmds@latest add qhjqhj00/financial-llm-zero-shot-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/financial-llm-zero-shot-eval/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-21
- Page: https://skillmd.com/skills/qhjqhj00/financial-llm-zero-shot-eval

---


# financial-llm-zero-shot-eval

> Zero is Not Hero Yet: Benchmarking Zero-Shot Performance of LLMs for Financial Tasks — Shah et al. (2023) (arXiv:2305.16633, 2023)

## What this evaluates

This benchmark evaluates the zero-shot instruction-following and classification capabilities of large language models on four financial NLP tasks: FOMC communication sentiment, general financial sentiment, numerical claim detection, and named entity recognition. It measures how well generative models can perform these tasks without fine-tuning compared to traditional PLMs.

## Datasets

- **Financial NLP Tasks (FOMC, Sentiment, Claim Detection, NER)** — total ?; splits: train (-1), test (-1), val (-1)

## Metrics

- `F1 score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall calculated over predicted vs. gold labels. For classification tasks, it is computed per instance or macro-averaged. For NER, it is computed over token-level BIO tags.

## Input / output format

**Input**: For classification tasks: A system prompt instructing the model to act as an expert classifier, followed by the target sentence. For NER: A tokenized sentence with instructions to assign BIO-style labels (Person, Location, Organisation, Other) in a 'token:label' format.

**Output**: Classification tasks: The predicted label ('HAWKISH'/'DOVISH'/'NEUTRAL', 'NEGATIVE'/'POSITIVE'/'NEUTRAL', 'INCLAIM'/'OUTOFCLAIM') on the first line, followed by a short explanation on the second line. NER: A list of 'token:label' pairs on separate lines.

## Scoring recipe

```python
def compute_f1(predictions, gold):
    pred_labels = [p.split('\n')[0].strip() for p in predictions]
    tp = sum(1 for p, g in zip(pred_labels, gold) if p == g)
    fp = sum(1 for p, g in zip(pred_labels, gold) if p != g)
    fn = sum(1 for p, g in zip(pred_labels, gold) if p != g)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
```

## Common pitfalls

- The zero-shot prompts enforce a strict two-line output format (label then explanation); models that output explanations first or add conversational filler will break the parser.
- NER uses a custom BIO tagging scheme with _B and _I suffixes that differ from standard CoNLL formats, requiring custom parsing logic.
- Evaluation is strictly zero-shot; any few-shot examples or chain-of-thought prompting will invalidate the benchmark's zero-shot claim.
- Temperature is fixed at 0.0, meaning results are deterministic and do not capture model variance or sampling effects.

## Evidence (verbatim from paper)

> If the validation F1 score doesn't improve by more than or equal to 1e-2 in the next 7 epochs then we use the best model stored earlier as the final fine-tuned model.

## Citation

```bibtex
@misc{shah2023zeronot,
  title={Zero is Not Hero Yet: Benchmarking Zero-Shot Performance of LLMs for Financial Tasks},
  author={Shah et al. (2023)},
  year={2023},
  note={arXiv:2305.16633}
}
```

- arXiv: 2305.16633