Finpt Eval

finpt-eval

FinPT: Financial Risk Prediction with Profile Tuning on Pretrained Foundation Models — Yin et al. (2023) (arXiv:2308.00065, 2023)

What this evaluates

Evaluates the ability of foundation models and tabular baselines to predict financial risk (default, fraud, churn) by classifying customer profiles generated from tabular data. It probes how well profile-based tuning captures richer customer semantics compared to isolated table-based classification.

Datasets

• FinBench — total 333000; splits: train (-1), val (-1), test (-1); repo https://github.com/YuweiYin/FinPT

Metrics

• F1-score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall). Calculated per class and averaged for binary classification tasks.

Input / output format

Input: Natural-language customer profiles (tabular-to-text transformations) generated via ChatGPT instructions, fed into pretrained foundation models or tabular baselines.

Output: Binary classification label indicating financial risk (e.g., default, fraud, or churn).

Scoring recipe

def compute_f1(y_true, y_pred):
    tp = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 1)
    fp = sum(1 for t, p in zip(y_true, y_pred) if t == 0 and p == 1)
    fn = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 0)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return f1

Common pitfalls

• Using accuracy instead of F1-score on imbalanced financial risk datasets leads to misleadingly high scores due to false negatives.
• Failing to average results over the four specified random seeds (0, 1, 42, 1234) will not match reported performance.
• Not using the validation set for checkpoint selection before testing on the held-out test set.

Evidence (verbatim from paper)

For all experiments on FinBench, we use F1-score as the evaluation metric since all datasets in FinBench are imbalanced binary classification task. It is more appropriate in this case than Accuracy because the latter may result in a high level of false negative.

Citation

@misc{yin2023finpt,
  title={FinPT: Financial Risk Prediction with Profile Tuning on Pretrained Foundation Models},
  author={Yin et al. (2023)},
  year={2023},
  note={arXiv:2308.00065}
}

• arXiv: 2308.00065