Multilingual Fraud Detection Eval

multilingual-fraud-detection-eval

Multilingual Financial Fraud Detection Using Machine Learning and Transformer Models: A Bangla-English Study — Uddin et al. (2026) (arXiv:2603.11358, 2026)

What this evaluates

Evaluates the ability of machine learning and transformer models to classify multilingual financial messages as legitimate or fraudulent. It probes handling of code-mixed Bangla-English text, low-resource language features, and structural indicators like URLs and phone numbers.

Datasets

• Financial scams detection dataset — total 523; splits: 5-fold stratified CV (523)

Metrics

• Accuracy (primary) — range: percent
  • Ratio of correctly classified instances to the total number of instances.
• Macro-averaged F1 score — range: percent
  • Unweighted mean of the F1 scores for the scam and ham classes, computed as the harmonic mean of precision and recall per class.
• PR-AUC — range: percent
  • Area under the Precision-Recall curve, measuring the model's ranking performance across all decision thresholds.

Input / output format

Input: Multilingual financial messages (text) containing Bangla, English, and code-mixed content.

Output: Binary label: 'ham' (legitimate) or 'scam' (fraudulent).

Scoring recipe

def compute_metrics(y_true, y_pred):
    tp = np.sum((y_true == 1) & (y_pred == 1))
    tn = np.sum((y_true == 0) & (y_pred == 0))
    fp = np.sum((y_true == 0) & (y_pred == 1))
    fn = np.sum((y_true == 1) & (y_pred == 0))
    accuracy = (tp + tn) / (tp + tn + fp + fn)
    prec_scam = tp / (tp + fp)
    rec_scam = tp / (tp + fn)
    f1_scam = 2 * prec_scam * rec_scam / (prec_scam + rec_scam)
    prec_ham = tn / (tn + fn)
    rec_ham = tn / (tn + fp)
    f1_ham = 2 * prec_ham * rec_ham / (prec_ham + rec_ham)
    macro_f1 = (f1_scam + f1_ham) / 2
    return accuracy, macro_f1

Common pitfalls

• The dataset is small (523 samples), leading to notable variance across cross-validation folds, especially for the transformer model.
• The transformer exhibits a strong bias toward predicting the scam class, resulting in high recall but elevated false positive rates compared to classical models.
• Code-mixing and limited Bangla financial vocabulary constrain both TF-IDF and multilingual transformer representations.

Evidence (verbatim from paper)

All models are evaluated using 5-fold stratified cross-validation to ensure robust performance estimation across different data partitions. We report accuracy, macro-averaged F1 score, and Precision-Recall Area Under Curve (PR-AUC) as primary evaluation metrics.

Citation

@misc{uddin2026multilingual,
  title={Multilingual Financial Fraud Detection Using Machine Learning and Transformer Models: A Bangla-English Study},
  author={Uddin et al. (2026)},
  year={2026},
  note={arXiv:2603.11358}
}

• arXiv: 2603.11358