Flare Es Eval

flare-es-eval

D'olares or Dollars? Unraveling the Bilingual Prowess of Financial LLMs Between Spanish and English — Xiao Zhang et al. (2024) (arXiv:2402.07405, 2024)

What this evaluates

Evaluates bilingual (Spanish-English) financial understanding, prediction, and generation capabilities of LLMs. Probes cross-lingual transfer, domain-specific instruction following, and performance disparity between high-resource and low-resource financial tasks.

Datasets

• FLARE-ES — total ?; splits: test (-1); repo https://github.com/chancefocus/PIXIU

Metrics

• Acc (primary) — range: [0, 1]
  • Accuracy: proportion of correctly predicted labels out of total instances.
• F1 (primary) — range: [0, 1]
  • F1 Score: harmonic mean of precision and recall, typically macro-averaged across classes.
• rouge1 — range: [0, 1]
  • ROUGE-1: unigram overlap between generated and reference text.
• rouge2 — range: [0, 1]
  • ROUGE-2: bigram overlap between generated and reference text.
• rougeL — range: [0, 1]
  • ROUGE-L: longest common subsequence overlap between generated and reference text.
• EntityF1 — range: [0, 1]
  • Entity F1: F1 score computed over extracted named entities.
• EmAcc — range: [0, 1]
  • Exact Match Accuracy: proportion of instances where the generated output exactly matches the reference.
• MCC — range: [-1, 1]
  • Matthews Correlation Coefficient: correlation coefficient between predicted and actual binary classifications.
• AvgF1 — range: [0, 1]
  • Average F1 score across multiple classes or tasks.

Input / output format

Input: Instruction prompts in Spanish or English containing financial domain questions, tasks, or text requiring model generation or classification.

Output: Model-generated text, class labels, or numerical answers depending on the specific task (e.g., classification labels, extracted entities, generated summaries, or predicted values).

Scoring recipe

def compute_metrics(predictions, references):
    acc = sum(p == r for p, r in zip(predictions, references)) / len(references)
    f1 = f1_score(references, predictions, average='macro')
    rouge = rouge_score(references, predictions, use_stemmer=True)
    em_acc = sum(p == r for p, r in zip(predictions, references)) / len(references)
    mcc = matthews_corrcoef(references, predictions)
    return {'Acc': acc, 'F1': f1, 'rouge1': rouge['rouge1'], 'rouge2': rouge['rouge2'], 'rougeL': rouge['rougeL'], 'EmAcc': em_acc, 'MCC': mcc}

Common pitfalls

• Models show significant performance drops on Spanish financial tasks compared to English due to data scarcity.
• Summarization tasks (e.g., FNS-2023, ECTSum) are highly challenging, often yielding near-zero scores across most models.
• Cross-lingual transfer does not automatically guarantee parity; specialized instruction tuning in the target language is required.

Evidence (verbatim from paper)

Table 3 presents a detailed comparative performance analysis of our FinMA-ES models against other leading large language models (LLMs) on the FLARE-ES benchmark. | Dataset | Metrics | ChatGPT | GPT4 | LLaMA2-7B | ... | MultiFin | Acc | F1 | ...

Citation

@misc{zhang2024dolares,
  title={D'olares or Dollars? Unraveling the Bilingual Prowess of Financial LLMs Between Spanish and English},
  author={Xiao Zhang et al. (2024)},
  year={2024},
  note={arXiv:2402.07405}
}

• arXiv: 2402.07405