Fingen Eval

fingen-eval

FinGen: A Dataset for Argument Generation in Finance — Chung-Chi Chen et al. (2024) (arXiv:2405.20708, 2024)

What this evaluates

Evaluates forward-looking argument generation in finance across text-to-claim, chart-to-argument, and news-to-argument tasks. Probes a model's ability to generate plausible, structured future scenarios and claims based on financial inputs while maintaining factual consistency and handling financial terminology and numerals.

Datasets

• FinGen — total ?; splits: train (-1), test (-1)

Metrics

• ROUGE-1 (primary) — range: [0, 1]
  • F1 score of unigram overlap between generated and reference text.
• ROUGE-2 — range: [0, 1]
  • F1 score of bigram overlap between generated and reference text.
• ROUGE-L — range: [0, 1]
  • F1 score based on the longest common subsequence between generated and reference text.
• BERT Score — range: [0, 1]
  • Cosine similarity between contextual embeddings of generated and reference text, averaged over tokens.

Input / output format

Input: Task-dependent: (1) Text excerpts (e.g., earnings calls), (2) Financial charts/images, (3) News headlines or full articles.

Output: Forward-looking financial claims or arguments, typically starting with phrases like 'We expect' or 'We believe', containing financial terms and numerals.

Scoring recipe

def compute_metrics(predictions, references):
    rouge = rouge_score(references, predictions, avg=True)  # ROUGE-1, 2, L
    bert = bert_score(references, predictions, lang='en')   # BERT Score F1
    return rouge, bert

Common pitfalls

• Models heavily copy financial terms from the input rather than generating novel ones.
• Models systematically under-generate numerals compared to ground truth and exhibit numeral hallucination.
• Generated claims often rely on generic boilerplate phrases like 'We expect' or 'We believe' without substantive reasoning.

Evidence (verbatim from paper)

We adopt ROUGE metrics and BERT Scores for evaluating all results.

Citation

@misc{chen2024fingen,
  title={FinGen: A Dataset for Argument Generation in Finance},
  author={Chung-Chi Chen et al. (2024)},
  year={2024},
  note={arXiv:2405.20708}
}

• arXiv: 2405.20708