Scenario Bias Financial Misinfo Eval

scenario-bias-financial-misinfo-eval

Same Claim, Different Judgment: Benchmarking Scenario-Induced Bias in Multilingual Financial Misinformation Detection — Liu et al. (2026) (arXiv:2601.05403, 2026)

What this evaluates

Evaluates how scenario-induced contextual factors (personality, region, identity) and multilingual settings alter LLM judgments on financial misinformation claims, quantifying behavioral bias as the performance shift relative to a neutral baseline.

Datasets

• Multilingual Financial Misinformation Dataset — total 144; splits: test (144); repo https://github.com/lzw108/FMD

Metrics

• Bias_scen (primary) — range: [0, 1]
  • Absolute difference in F1 scores between scenario-conditioned predictions and base (unconditioned) predictions against ground truth: |F1(l_scen, l_gold) - F1(l_base, l_gold)|.

Input / output format

Input: A financial misinformation claim combined with a contextual scenario prompt (specifying role, personality/region/identity) or a neutral prompt without scenario context.

Output: A binary truthfulness label: True or False.

Scoring recipe

f1_scen = f1_score(gold, predictions_scen, average='binary')
f1_base = f1_score(gold, predictions_base, average='binary')
bias_scen = abs(f1_scen - f1_base)

Common pitfalls

• Calculating bias as absolute F1 under scenario rather than the difference from the base prompt.
• Using accuracy instead of F1, which can be misleading for imbalanced true/false distributions in financial news.
• Ignoring translation artifacts in non-English splits that may artificially inflate or deflate bias scores.

Evidence (verbatim from paper)

Given a piece of financial information claim $c$, the task is to determine the truthfulness label ($l_{scen},l_{base},l_{gold}\in L={True,False}$) of the claim $c$ in the scenario $s$. ... $\text{Bias}{\text{scen}}=|\text{F1}(l{\text{scen}},l_{\text{gold}})-\text{F1}(l_{\text{base}},l_{\text{gold}})|$ ... $l_{\text{scen}}$ denotes the LLMs’ predictions under specific financial scenarios, $l_{\text{base}}$ denotes their predictions without financial scenario information, and $l_{\text{gold}}$ represents the ground-truth labels. The behavioral bias is quantified as the difference in F1 scores between these two cases, reflecting how scenario context changes verification performance for the same claim.

Citation

@misc{liu2026sameclaim,
  title={Same Claim, Different Judgment: Benchmarking Scenario-Induced Bias in Multilingual Financial Misinformation Detection},
  author={Liu et al. (2026)},
  year={2026},
  note={arXiv:2601.05403}
}

• arXiv: 2601.05403