Financial LLM Zero Shot Eval

financial-llm-zero-shot-eval

Zero is Not Hero Yet: Benchmarking Zero-Shot Performance of LLMs for Financial Tasks — Shah et al. (2023) (arXiv:2305.16633, 2023)

What this evaluates

This benchmark evaluates the zero-shot instruction-following and classification capabilities of large language models on four financial NLP tasks: FOMC communication sentiment, general financial sentiment, numerical claim detection, and named entity recognition. It measures how well generative models can perform these tasks without fine-tuning compared to traditional PLMs.

Datasets

• Financial NLP Tasks (FOMC, Sentiment, Claim Detection, NER) — total ?; splits: train (-1), test (-1), val (-1)

Metrics

• F1 score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall calculated over predicted vs. gold labels. For classification tasks, it is computed per instance or macro-averaged. For NER, it is computed over token-level BIO tags.

Input / output format

Input: For classification tasks: A system prompt instructing the model to act as an expert classifier, followed by the target sentence. For NER: A tokenized sentence with instructions to assign BIO-style labels (Person, Location, Organisation, Other) in a 'token:label' format.

Output: Classification tasks: The predicted label ('HAWKISH'/'DOVISH'/'NEUTRAL', 'NEGATIVE'/'POSITIVE'/'NEUTRAL', 'INCLAIM'/'OUTOFCLAIM') on the first line, followed by a short explanation on the second line. NER: A list of 'token:label' pairs on separate lines.

Scoring recipe

def compute_f1(predictions, gold):
    pred_labels = [p.split('\n')[0].strip() for p in predictions]
    tp = sum(1 for p, g in zip(pred_labels, gold) if p == g)
    fp = sum(1 for p, g in zip(pred_labels, gold) if p != g)
    fn = sum(1 for p, g in zip(pred_labels, gold) if p != g)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

Common pitfalls

• The zero-shot prompts enforce a strict two-line output format (label then explanation); models that output explanations first or add conversational filler will break the parser.
• NER uses a custom BIO tagging scheme with _B and _I suffixes that differ from standard CoNLL formats, requiring custom parsing logic.
• Evaluation is strictly zero-shot; any few-shot examples or chain-of-thought prompting will invalidate the benchmark's zero-shot claim.
• Temperature is fixed at 0.0, meaning results are deterministic and do not capture model variance or sampling effects.

Evidence (verbatim from paper)

If the validation F1 score doesn't improve by more than or equal to 1e-2 in the next 7 epochs then we use the best model stored earlier as the final fine-tuned model.

Citation

@misc{shah2023zeronot,
  title={Zero is Not Hero Yet: Benchmarking Zero-Shot Performance of LLMs for Financial Tasks},
  author={Shah et al. (2023)},
  year={2023},
  note={arXiv:2305.16633}
}

• arXiv: 2305.16633