Sfiog Eval

sfiog-eval

Beyond Classification: Financial Reasoning in State-of-the-Art Language Models — Son et al. (2023) (arXiv:2305.01505, 2023)

What this evaluates

Evaluates large language models' ability to generate coherent, logically structured financial investment opinions based on company context and questions. It probes reasoning over mere knowledge retrieval by testing performance across varying degrees of familiarity and novelty in companies and questions.

Datasets

• sFIOG — total 11802; splits: test (-1); repo https://github.com/guijinSON/FIOG

Metrics

• ROUGE-L (primary) — range: [0, 1]
  • Recall-Oriented Understudy for Gisting Evaluation using the longest common subsequence. Measures the overlap of the longest matching word sequence between the generated text and the reference answer.
• BERTScore — range: [0, 1]
  • Computes semantic similarity by matching tokens in the generated and reference texts using contextual embeddings from a pre-trained BERT model, then calculating precision, recall, and F1 scores.
• ROUGE-2 — range: [0, 1]
  • Measures the overlap of 2-grams (bigrams) between the generated text and the reference answer.

Input / output format

Input: Company context (provided as full-text or Q&A format) paired with a specific question requiring an investment opinion.

Output: A generated investment opinion/thesis text, constrained to a maximum of 512 new tokens.

Scoring recipe

def compute_metrics(predictions, references):
    rouge_l = rouge_score(references, predictions, rouge_types=['rougeL'])['rougeL.fmeasure']
    rouge_2 = rouge_score(references, predictions, rouge_types=['rouge2'])['rouge2.fmeasure']
    bert_f1 = bert_score.score(predictions, references, lang='en')[2]
    return {
        'ROUGE-L': rouge_l,
        'ROUGE-2': rouge_2,
        'BERTScore': bert_f1.mean()
    }

Common pitfalls

• Models are constrained to exactly 512 max new tokens during decoding to prevent longer outputs from artificially inflating overlap metrics.
• LLM-based automated evaluators were explicitly excluded because they were found to misalign with human judgments.
• Type#2 questions (known companies, novel Q&A pairs) proved harder for models than Type#3 (novel companies), contrary to initial assumptions about knowledge generalization.

Evidence (verbatim from paper)

Automated metrics included rouge-2 and rougeL [Lin, 2004], measuring text overlap, and BERTScore [Zhang et al., 2019], assessing semantic similarity. ... LLama demonstrates superior performance, achieving the highest average scores in ROUGE-L (0.217) and BERTScore (0.821).

Citation

@misc{son2023beyondclassification,
  title={Beyond Classification: Financial Reasoning in State-of-the-Art Language Models},
  author={Son et al. (2023)},
  year={2023},
  note={arXiv:2305.01505}
}

• arXiv: 2305.01505