Financial Esg Nlp Bench Eval

financial-esg-nlp-bench-eval

SusGen-GPT: A Data-Centric LLM for Financial NLP and Sustainability Report Generation — Qilong Wu et al. (arXiv:2412.10906, 2024)

What this evaluates

Evaluates large language models on a suite of 13 financial and ESG-related NLP tasks, including sentiment analysis, classification, named entity recognition, relation extraction, financial question answering, text summarization, and sustainability report generation. It probes the model's ability to perform domain-specific reasoning, information extraction, and structured text generation in the financial sector.

Datasets

• FiQASA — total ?; splits: test (-1)
• FOMC — total ?; splits: test (-1)
• MultiFin — total ?; splits: test (-1)
• MLESG — total ?; splits: test (-1)
• NER — total ?; splits: test (-1)
• FINER-ORD — total ?; splits: test (-1)
• FinRED — total ?; splits: test (-1)
• SC — total ?; splits: test (-1)
• FinQA — total ?; splits: test (-1)
• TATQA — total ?; splits: test (-1)
• ConvFinQA — total ?; splits: test (-1)
• EDTSUM — total ?; splits: test (-1)
• TCFD-Bench — total ?; splits: test (-1)

Metrics

• F1 (primary) — range: [0, 1]
  • Harmonic mean of precision and recall. Computed macro-averaged for classification and relation extraction tasks.
• MicroF1 (primary) — range: [0, 1]
  • F1 score calculated by aggregating true positives, false positives, and false negatives across all classes before computing precision and recall.
• EntityF1 — range: [0, 1]
  • F1 score computed over extracted entity spans, requiring exact boundary and type matching.
• EmAcc (primary) — range: [0, 1]
  • Exact match accuracy; the model's predicted answer must exactly match the gold answer string.
• Rouge-1 — range: [0, 1]
  • Unigram overlap between generated and reference text, measured as F1 score.
• BertScore — range: [0, 1]
  • Cosine similarity between contextual embeddings (BERT) of generated and reference tokens, averaged across tokens.
• Rouge-L (primary) — range: [0, 1]
  • Longest common subsequence overlap between generated and reference text, measured as F1 score.
• METEOR — range: [0, 1]
  • Weighted F-score incorporating exact match, synonymy, stemming, and paraphrase matching.
• BLEU-1 — range: [0, 1]
  • Unigram precision with a brevity penalty to discourage overly short generations.

Input / output format

Input: Alpaca prompt template combining instruction, context, and question. For retrieval-augmented generation, top-10 relevant text chunks are retrieved using all-mpnet-base-v2 embeddings.

Output: Model-generated text: class labels, numerical answers, entity/relation spans, summaries, or structured TCFD-format ESG reports.

Scoring recipe

def evaluate(predictions, golds, task):
    if task in ['SA', 'RE']:
        return f1_score(golds, predictions, average='macro')
    elif task in ['HC', 'MLESG']:
        return f1_score(golds, predictions, average='micro')
    elif task == 'NER':
        return entity_f1(golds, predictions)
    elif task in ['FinQA', 'TATQA', 'ConvFinQA']:
        return exact_match(golds, predictions)
    elif task == 'SUM':
        return rouge1(golds, predictions), bertscore(golds, predictions)
    elif task == 'SRG':
        return rouge_l(golds, predictions), bertscore(golds, predictions), meteor(golds, predictions), bleu1(golds, predictions)

Common pitfalls

• Different tasks use fundamentally different metrics (e.g., F1 vs EmAcc vs Rouge), making cross-task averaging invalid.
• TCFD-Bench is a newly proposed benchmark without a public HuggingFace ID or established baseline code, requiring custom evaluation scripts.
• Evaluation is conducted in zero-shot or few-shot mode despite the model being fine-tuned on SusGen-30K, so results reflect adaptation capability rather than raw base performance.

Evidence (verbatim from paper)

We evaluate and compare SusGen-GPT on eight tasks in total against other baseline models, including seven financial NLP tasks using well-established benchmarks as well sustainability report generation (SRG) using our proposed TCFD-Bench. In SA, it achieves an F1 score of 0.72 on the FiQASA dataset... For FinQA and FinTQA, the model scores 0.57 on FinQA... SusGen-GPT outperformed ChatReport in Rouge-L (0.20 vs. 0.14), BERTScore (0.40 vs. 0.32), and METEOR (0.27 vs. 0.12)...

Citation

@misc{wu2024susgengpt,
  title={SusGen-GPT: A Data-Centric LLM for Financial NLP and Sustainability Report Generation},
  author={Qilong Wu et al.},
  year={2024},
  note={arXiv:2412.10906}
}

• arXiv: 2412.10906