Finlmeval Eval

finlmeval-eval

Is ChatGPT a Financial Expert? Evaluating Language Models on Financial Natural Language Processing — Guo et al. (2023) (arXiv:2310.12664, 2023)

What this evaluates

Evaluates the financial natural language processing capabilities of encoder-only and decoder-only language models across multiple classification tasks. It probes zero-shot prompting, in-context learning strategies, and the impact of data availability (public vs. proprietary) on model performance.

Datasets

• FinSent — total ?; splits: train (-1), val (-1), test (-1)
• FPB — total ?; splits: train (-1), val (-1), test (-1)
• FiQA SA — total ?; splits: train (-1), val (-1), test (-1)
• ESG — total ?; splits: train (-1), val (-1), test (-1)
• FLS — total ?; splits: train (-1), val (-1), test (-1)
• QA — total ?; splits: train (-1), val (-1), test (-1)
• Headlines-PDU — total ?; splits: train (-1), val (-1), test (-1)
• Headlines-PDC — total ?; splits: train (-1), val (-1), test (-1)
• Headlines-PDD — total ?; splits: train (-1), val (-1), test (-1)
• Headlines-PI — total ?; splits: train (-1), val (-1), test (-1)
• Headlines-AC — total ?; splits: train (-1), val (-1), test (-1)
• Headlines-FI — total ?; splits: train (-1), val (-1), test (-1)
• Headlines-PS — total ?; splits: train (-1), val (-1), test (-1)
• NER — total ?; splits: train (-1), val (-1), test (-1)
• FOMC — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Not explicitly named in the text; values in [0,1] suggest Accuracy or F1-score. Computed as the proportion of correctly predicted labels over total instances per task.

Input / output format

Input: Text snippets (sentences or headlines) for classification tasks. For in-context learning, prompts are constructed by appending selected demonstration examples (random or similarity-based) to the query.

Output: Predicted class label.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)

Common pitfalls

• Data leakage risks on proprietary datasets (FinSent, ESG, FLS, QA) can artificially inflate zero-shot LLM performance if not strictly controlled.
• In-context learning effectiveness is highly prompt-dependent; random vs. similarity-based demonstration selection yields divergent results across tasks.
• Assuming decoder-only LLMs universally outperform fine-tuned encoder-only models ignores task-specific data scarcity and imbalance issues.

Evidence (verbatim from paper)

Table 2 compares the results of the fine-tuned encoder-only models and zero-shot decoder-only models in 9 financial datasets. We have the following findings: In 6 out of 9 datasets, fine-tuned encoder-only models can perform better than decoder-only models.

Citation

@misc{guo2023ischatgpt,
  title={Is ChatGPT a Financial Expert? Evaluating Language Models on Financial Natural Language Processing},
  author={Guo et al. (2023)},
  year={2023},
  note={arXiv:2310.12664}
}

• arXiv: 2310.12664