Mfmdqwen Eval

mfmdqwen-eval

MFMDQwen: Multilingual Financial Misinformation Detection Based on Large Language Model — Liu et al. (2026) (arXiv:2604.18272, 2026)

What this evaluates

Evaluates large language models on multilingual financial misinformation detection across nine tasks in English, Chinese, Greek, and Bengali. It probes the model's ability to identify false or misleading financial claims, handle numerical sensitivity and reversed causality, and generalize across diverse linguistic settings.

Datasets

• MFMDBench — total ?; splits: test (-1); repo https://github.com/lzw108/FMD

Metrics

• Accuracy — range: [0, 1]
  • Standard classification accuracy, calculated as the number of correct predictions divided by the total number of predictions.
• Macro-F1 (primary) — range: [0, 1]
  • Macro-averaged F1 score, computed as the unweighted mean of the F1 scores for each class, treating all classes equally regardless of support.

Input / output format

Input: Multilingual textual financial claims or news snippets (English, Chinese, Greek, or Bengali), often accompanied by evidence or context.

Output: Binary classification label indicating whether the input contains financial misinformation.

Scoring recipe

def compute_metrics(predictions, gold_labels):
    accuracy = sum(p == g for p, g in zip(predictions, gold_labels)) / len(gold_labels)
    macro_f1 = 0.0
    for label in unique_labels:
        tp = sum(1 for p, g in zip(predictions, gold_labels) if p == label and g == label)
        fp = sum(1 for p, g in zip(predictions, gold_labels) if p == label and g != label)
        fn = sum(1 for p, g in zip(predictions, gold_labels) if p != label and g == label)
        precision = tp / (tp + fp) if (tp + fp) > 0 else 0
        recall = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
        macro_f1 += f1
    macro_f1 /= len(unique_labels)
    return accuracy, macro_f1

Common pitfalls

• Models with strict safety mechanisms may refuse to answer or output irrelevant responses, artificially lowering F1 scores.
• Reasoning-capable models may underperform non-reasoning counterparts of the same size due to overthinking or generating incorrect intermediate steps.
• Performance on low-resource languages (e.g., Bengali) may appear near-saturated due to limited dataset size and reduced task complexity rather than true model capability.

Evidence (verbatim from paper)

We uses metrics such as Accuracy, Macro-F1 for misinformation detection evaluation. In the following analysis, we primarily focus on the F1 score.

Citation

@misc{liu2026mfmdqwen,
  title={MFMDQwen: Multilingual Financial Misinformation Detection Based on Large Language Model},
  author={Liu et al. (2026)},
  year={2026},
  note={arXiv:2604.18272}
}

• arXiv: 2604.18272