Golden Touchstone Eval

golden-touchstone-eval

Golden Touchstone: A Comprehensive Bilingual Benchmark for Evaluating Financial Large Language Models — Wu et al. (2024) (arXiv:2411.06272, 2024)

What this evaluates

This benchmark evaluates the capability of large language models to perform a wide range of financial natural language processing tasks in both English and Chinese. It probes domain-specific understanding, information extraction, reasoning, and generation across sentiment analysis, classification, entity/relation extraction, summarization, question answering, and stock movement prediction.

Datasets

• FPB — total ?; splits: test (-1); repo https://github.com/IDEA-FinAI/Golden-Touchstone
• Fiqa-SA — total ?; splits: test (-1); repo https://github.com/IDEA-FinAI/Golden-Touchstone
• Headlines — total ?; splits: test (-1); repo https://github.com/IDEA-FinAI/Golden-Touchstone
• FOMC — total ?; splits: test (-1); repo https://github.com/IDEA-FinAI/Golden-Touchstone
• lendingclub — total ?; splits: test (-1); repo https://github.com/IDEA-FinAI/Golden-Touchstone
• NER — total ?; splits: test (-1); repo https://github.com/IDEA-FinAI/Golden-Touchstone
• FinRE — total ?; splits: test (-1); repo https://github.com/IDEA-FinAI/Golden-Touchstone
• CFA — total ?; splits: test (-1); repo https://github.com/IDEA-FinAI/Golden-Touchstone
• EDTSUM — total ?; splits: test (-1); repo https://github.com/IDEA-FinAI/Golden-Touchstone
• Finqa — total ?; splits: test (-1); repo https://github.com/IDEA-FinAI/Golden-Touchstone
• Convfinqa — total ?; splits: test (-1); repo https://github.com/IDEA-FinAI/Golden-Touchstone
• DJIA — total ?; splits: test (-1); repo https://github.com/IDEA-FinAI/Golden-Touchstone
• FinFe-CN — total ?; splits: test (-1); repo https://github.com/IDEA-FinAI/Golden-Touchstone
• FinNL-CN — total ?; splits: test (-1); repo https://github.com/IDEA-FinAI/Golden-Touchstone
• FinESE-CN — total ?; splits: test (-1); repo https://github.com/IDEA-FinAI/Golden-Touchstone
• FinRE-CN — total ?; splits: test (-1); repo https://github.com/IDEA-FinAI/Golden-Touchstone
• FinQa-CN — total ?; splits: test (-1); repo https://github.com/IDEA-FinAI/Golden-Touchstone

Metrics

• Weighted-F1 (primary) — range: [0, 1]
  • Harmonic mean of precision and recall, averaged across classes weighted by their support (frequency). Standard for imbalanced financial classification tasks.
• ACC — range: [0, 1]
  • Accuracy, calculated as the number of correct predictions divided by the total number of predictions.
• MCC — range: [-1, 1]
  • Matthews Correlation Coefficient, measuring the quality of binary/multiclass classifications. Returns a value between -1 and +1.
• Entity-F1 — range: [0, 1]
  • F1 score for Named Entity Recognition, evaluating exact match of extracted entity spans against gold labels.
• Relation-F1 — range: [0, 1]
  • F1 score for Relation Extraction, evaluating exact match of predicted relation triples (head, relation, tail) against gold labels.
• Rouge-1/2/L — range: [0, 1]
  • Recall-Oriented Understudy for Gisting Evaluation. Measures overlap of unigrams (Rouge-1), bigrams (Rouge-2), and longest common subsequence (Rouge-L) between generated and reference summaries.
• BLEU — range: [0, 1]
  • Bilingual Evaluation Understudy. Measures n-gram precision of the generated text against references with a brevity penalty.
• RMACC — range: [0, 1]
  • Relative accuracy metric for financial question answering, comparing predicted numerical/label answers against gold standards.
• ORMACC — range: [0, 1]
  • Chinese variant of RMACC used for Chinese financial QA and entity extraction tasks, measuring relative accuracy against gold standards.

Input / output format

Input: Financial text inputs including news articles, research reports, financial statements, questions, and prompts for classification, extraction, summarization, or prediction tasks. Inference uses greedy decoding with fixed PyTorch and CUDA random seeds.

Output: Model-generated text or discrete labels depending on the task: class labels, extracted entity/relation spans, summary paragraphs, numerical answers, or stock movement predictions (up/down).

Scoring recipe

def compute_metrics(predictions, golds, task):
    if task in ['SentimentAnalysis', 'Classification', 'StockMovementPrediction']:
        return accuracy_score(golds, predictions), f1_score(golds, predictions, average='weighted')
    elif task == 'lendingclub':
        return matthews_corrcoef(golds, predictions)
    elif task in ['EntityExtraction', 'RelationExtraction']:
        return exact_match_f1(predictions, golds)
    elif task == 'Summarization':
        return rouge_score(golds, predictions, types=['rouge1','rouge2','rougeL']), bleu_score(golds, predictions)
    elif task in ['QuestionAnswering']:
        return relative_accuracy(predictions, golds)
    return 0.0

Common pitfalls

• Zero-shot inference on highly specialized datasets (e.g., LendingClub for risk control) yields poor results without domain-specific instruction fine-tuning.
• Stock movement prediction tasks in this benchmark rely solely on textual news data, ignoring volume-price and factor analysis, making them inherently difficult for LLMs.
• Summarization metrics (BLEU/Rouge) are consistently low across models due to the complexity and specificity of financial text, not necessarily indicating model failure.

Evidence (verbatim from paper)

GPT-4o consistently performs well in tasks like Sentiment Analysis with a Weighted-F1 score of 0.8084 and an accuracy of 0.8093 on the FPB dataset and Multiple Choice, showcasing robustness in understanding sentiment-based and structured questions. However, it struggles significantly in Relation Extraction achieving a Relation-F1 score of 0.1613 on the FinRE dataset and Entity Extraction with an Entity-F1 score of 0.1800 on the NER dataset, suggesting difficulties with detailed information extraction and handling relationships.

Citation

@misc{wu2024goldentouchstone,
  title={Golden Touchstone: A Comprehensive Bilingual Benchmark for Evaluating Financial Large Language Models},
  author={Wu et al. (2024)},
  year={2024},
  note={arXiv:2411.06272}
}

• arXiv: 2411.06272