Ice Flare Eval

ice-flare-eval

No Language is an Island: Unifying Chinese and English in Financial Large Language Models, Instruction Data, and Benchmarks — Gang Hu et al. (2024) (arXiv:2403.06249, 2024)

What this evaluates

Evaluates bilingual (Chinese and English) financial large language models across 14 NLP tasks, including sentiment analysis, classification, question answering, and information extraction. It probes cross-lingual adaptability, domain-specific reasoning, and instruction-following capabilities on financial text.

Datasets

• FE — total 2020; splits: test (2020); repo https://github.com/The-FinAI/PIXIU
• StockB — total 1962; splits: test (1962); repo https://github.com/The-FinAI/PIXIU
• CFPB — total 970; splits: test (970); repo https://github.com/The-FinAI/PIXIU
• CFiQA-SA — total 233; splits: test (233); repo https://github.com/The-FinAI/PIXIU
• FPB — total 970; splits: test (970); repo https://github.com/The-FinAI/PIXIU
• FiQA-SA — total 235; splits: test (235); repo https://github.com/The-FinAI/PIXIU
• Corpus — total 10000; splits: test (10000); repo https://github.com/The-FinAI/PIXIU
• AFQMC — total 4316; splits: test (4316); repo https://github.com/The-FinAI/PIXIU
• NL — total 884; splits: test (884); repo https://github.com/The-FinAI/PIXIU
• NL2 — total 884; splits: test (884); repo https://github.com/The-FinAI/PIXIU
• NSP — total 500; splits: test (500); repo https://github.com/The-FinAI/PIXIU
• FinevalF — total 222; splits: test (222); repo https://github.com/The-FinAI/PIXIU
• StcokA — total 1477; splits: test (1477); repo https://github.com/The-FinAI/PIXIU
• CACL18 — total 511; splits: test (511); repo https://github.com/The-FinAI/PIXIU
• CBigData18 — total 159; splits: test (159); repo https://github.com/The-FinAI/PIXIU
• CIKM18 — total 86; splits: test (86); repo https://github.com/The-FinAI/PIXIU
• ACL18 — total 3720; splits: test (3720); repo https://github.com/The-FinAI/PIXIU
• BigData18 — total 1472; splits: test (1472); repo https://github.com/The-FinAI/PIXIU
• RE — total 1489; splits: test (1489); repo https://github.com/The-FinAI/PIXIU
• CHeadlines — total 2051; splits: test (2051); repo https://github.com/The-FinAI/PIXIU
• Headlines — total 20547; splits: test (20547); repo https://github.com/The-FinAI/PIXIU
• German — total 200; splits: test (200); repo https://github.com/The-FinAI/PIXIU
• Australian — total 139; splits: test (139); repo https://github.com/The-FinAI/PIXIU
• FOMC — total 496; splits: test (496); repo https://github.com/The-FinAI/PIXIU
• QA — total 2469; splits: test (2469); repo https://github.com/The-FinAI/PIXIU
• CEnQA — total 133; splits: test (133); repo https://github.com/The-FinAI/PIXIU
• CConFinQA — total 237; splits: test (237); repo https://github.com/The-FinAI/PIXIU
• EnQA — total 1147; splits: test (1147); repo https://github.com/The-FinAI/PIXIU
• ConFinQA — total 1490; splits: test (1490); repo https://github.com/The-FinAI/PIXIU
• CNER — total 337; splits: test (337); repo https://github.com/The-FinAI/PIXIU
• NER — total 98; splits: test (98); repo https://github.com/The-FinAI/PIXIU
• FINER-ORD — total 1075; splits: test (1075); repo https://github.com/The-FinAI/PIXIU
• 19CCKS — total 2936; splits: test (2936); repo https://github.com/The-FinAI/PIXIU
• 20CCKS — total 9159; splits: test (9159); repo https://github.com/The-FinAI/PIXIU
• 21CCKS — total 1400; splits: test (1400); repo https://github.com/The-FinAI/PIXIU
• 22CCKS — total 11829; splits: test (11829); repo https://github.com/The-FinAI/PIXIU
• NA — total 3600; splits: test (3600); repo https://github.com/The-FinAI/PIXIU
• ECTSUM — total 495; splits: test (495); repo https://github.com/The-FinAI/PIXIU
• EDTSUM — total 2000; splits: test (2000); repo https://github.com/The-FinAI/PIXIU

Metrics

• F1 Accuracy (primary) — range: [0, 1]
  • Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall). Computed per class and macro-averaged for multi-class tasks.
• MCC — range: [-1, 1]
  • Matthews Correlation Coefficient, measuring binary/multiclass classification quality considering true/false positives and negatives.
• EM Accuracy — range: [0, 1]
  • Exact match accuracy; counts a prediction as correct only if it exactly matches the gold answer string.
• Entity F1 — range: [0, 1]
  • F1 score computed over extracted entity spans and types.
• ROUGE/BERTScore/BARTScore — range: [0, 1]
  • ROUGE measures n-gram overlap; BERTScore and BARTScore use contextual embeddings for semantic similarity.

Input / output format

Input: Financial text, news articles, stock data, or questions in Chinese or English, formatted as zero-shot or few-shot instruction prompts.

Output: Predicted class labels, extracted entity sequences, generated answers, or text summaries, depending on the specific task.

Scoring recipe

def compute_metric(predictions, gold, metric_type):
    if metric_type == 'F1 Accuracy':
        tp = sum(1 for p, g in zip(predictions, gold) if p == g)
        prec = tp / len(predictions) if predictions else 0
        rec = tp / len(gold) if gold else 0
        return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    elif metric_type == 'EM Accuracy':
        return sum(1 for p, g in zip(predictions, gold) if p == g) / len(gold)
    elif metric_type == 'MCC':
        return matthews_corrcoef(gold, predictions)
    # For ROUGE/BERTScore/BARTScore, use standard library implementations
    # For Entity F1, compute precision/recall over span matches

Common pitfalls

• Confusing language-specific test splits (e.g., FinSP has separate zh and en sets with different sizes and sources).
• Generation tasks (FinST, FinER) require strict label sequence formatting; models often fail to output exact token sequences without careful prompt design.
• Using translated training data (DTT) for evaluation instead of original held-out test sets, which artificially inflates cross-lingual performance.

Evidence (verbatim from paper)

Table 2: The details of specific task, evaluation metric, language type, dataset name and test size in ICE-FLARE. ... FinSA | F1 Accuracy | zh | FE StockB CFPB CFiQA-SA en | 2,020 1,962 970 233

Citation

@misc{hu2024no,
  title={No Language is an Island: Unifying Chinese and English in Financial Large Language Models, Instruction Data, and Benchmarks},
  author={Gang Hu et al. (2024)},
  year={2024},
  note={arXiv:2403.06249}
}

• arXiv: 2403.06249