Financial Nlp Eval

financial-nlp-eval

Are ChatGPT and GPT-4 General-Purpose Solvers for Financial Text Analytics? A Study on Several Typical Tasks — Li et al. (2023) (arXiv:2305.05862, 2023)

What this evaluates

Evaluates the capability of large language models (ChatGPT and GPT-4) and domain-specific models to solve a variety of financial text analytics tasks. It probes performance across sentiment analysis, classification, information extraction, and question answering, measuring how well models handle domain-specific knowledge and structured prediction.

Datasets

• Financial NLP Tasks (Sentiment, Classification, NER, RE, QA) — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correct predictions out of total instances.
• macro-F1 score — range: [0, 1]
  • Unweighted mean of per-class F1 scores, treating all classes equally regardless of support.
• weighted F1 score — range: [0, 1]
  • Mean of per-class F1 scores weighted by the number of true instances for each class.
• entity-level F1 score — range: [0, 1]
  • F1 score calculated at the entity span level for NER, requiring exact match of predicted and gold entity boundaries and types.

Input / output format

Input: Raw financial text, news, social media posts, filings, or reports. For QA, questions with context. Prompts are formatted for zero-shot, few-shot, or Chain-of-Thought (CoT) inference.

Output: Model-generated predictions: class labels, sentiment scores, entity spans, relation triples, or textual answers.

Scoring recipe

def compute_metrics(predictions, golds):
    acc = sum(p == g for p, g in zip(predictions, golds)) / len(golds)
    # Compute precision, recall, F1 per class
    # macro-F1 = mean of per-class F1s
    # weighted-F1 = sum(support_i * F1_i) / total_samples
    # For NER: count exact span matches for precision/recall
    return acc, macro_f1, weighted_f1

Common pitfalls

• The paper evaluates multiple distinct task types but does not specify a single headline metric; accuracy, macro-F1, and weighted-F1 are all reported without clear hierarchy.
• Dataset names, exact sizes, and train/val/test splits are not provided in the main text and are deferred to Table 1 and Appendix B, making independent replication difficult.

Evidence (verbatim from paper)

We use accuracy, macro-F1 score, and weighted F1 score (Wu et al., 2023) as the evaluation metrics. For the NER task, we calculate the entity-level F1 score.

Citation

@misc{li2023arechatgpt,
  title={Are ChatGPT and GPT-4 General-Purpose Solvers for Financial Text Analytics? A Study on Several Typical Tasks},
  author={Li et al. (2023)},
  year={2023},
  note={arXiv:2305.05862}
}

• arXiv: 2305.05862