Financial Misinformation Detection Eval

financial-misinformation-detection-eval

Fact4ac at the Financial Misinformation Detection Challenge Task: Reference-Free Financial Misinformation Detection via Fine-Tuning and Few-Shot Prompting of Large Language Models — Hoang et al. (2026) (arXiv:2604.14640, 2026)

What this evaluates

This benchmark evaluates a model's ability to detect financial misinformation in a reference-free setting, where models must classify financial narrative paragraphs as true or false without external knowledge or source documents. It probes semantic pattern recognition for financial manipulation, omission detection, and domain-specific generalization.

Datasets

• MisD@ICWSM2026 — total ?; splits: development (-1), public test (-1), private test (-1)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Proportion of correctly classified instances out of the total number of instances. Calculated as (True Positives + True Negatives) / Total.
• F1 — range: [0, 1]
  • Harmonic mean of Precision and Recall. Calculated as 2 * (Precision * Recall) / (Precision + Recall).
• Precision — range: [0, 1]
  • Ratio of true positive predictions to all positive predictions. Calculated as TP / (TP + FP).
• Recall — range: [0, 1]
  • Ratio of true positive predictions to all actual positives. Calculated as TP / (TP + FN).

Input / output format

Input: A single financial narrative paragraph. No external references, source documents, or factual grounding materials are provided (reference-free setting).

Output: Binary classification label: 'True' (legitimate financial narrative) or 'False' (misinformation/manipulated narrative).

Scoring recipe

def compute_metrics(predictions, gold_labels):
    tp = sum(1 for p, g in zip(predictions, gold_labels) if p == 'True' and g == 'True')
    tn = sum(1 for p, g in zip(predictions, gold_labels) if p == 'False' and g == 'False')
    fp = sum(1 for p, g in zip(predictions, gold_labels) if p == 'True' and g == 'False')
    fn = sum(1 for p, g in zip(predictions, gold_labels) if p == 'False' and g == 'True')
    total = len(gold_labels)
    accuracy = (tp + tn) / total
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    return {'accuracy': accuracy, 'precision': precision, 'recall': recall, 'f1': f1}

Common pitfalls

• Fine-tuning exclusively on negative (False) samples introduces bias and degrades few-shot prompting performance; balanced training data is required for optimal results.
• Pretrained LLMs without domain-specific fine-tuning perform near random chance (~50%) on this task, regardless of model scale or prompting strategy.
• Public and private test sets evaluate different generalization capabilities; private set performance is the stricter benchmark for real-world robustness and calibration.

Evidence (verbatim from paper)

On the Public test set, our team demonstrated a robust capability in detecting misinformation with an Accuracy of 95.4% and a dominant F1-score of 95.4%.

Citation

@misc{hoang2026fact4ac,
  title={Fact4ac at the Financial Misinformation Detection Challenge Task: Reference-Free Financial Misinformation Detection via Fine-Tuning and Few-Shot Prompting of Large Language Models},
  author={Hoang et al. (2026)},
  year={2026},
  note={arXiv:2604.14640}
}

• arXiv: 2604.14640