Folktexts Eval

folktexts-eval

Evaluating language models as risk scores — Cruz et al. (2024) (arXiv:2407.14614, 2024)

What this evaluates

Evaluates the calibration and predictive accuracy of language models when used as risk scorers for tabular prediction tasks. It probes whether models can accurately quantify outcome uncertainty (calibration) while maintaining discriminative power (AUC) on natural-language versions of tabular datasets.

Datasets

• folktexts — total ?; splits: test (-1); repo https://github.com/socialfoundations/folktexts

Metrics

• ECE (primary) — range: [0, 1]
  • Expected Calibration Error. Measures the difference between predicted confidence and actual accuracy across probability bins. Lower is better.
• Brier score — range: [0, 1]
  • Mean squared difference between predicted probability and actual outcome (0 or 1). Lower is better.
• AUC — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve. Measures the model's ability to discriminate between positive and negative classes. Higher is better.
• Accuracy — range: [0, 1]
  • Proportion of correctly classified instances. Higher is better.

Input / output format

Input: Natural-language descriptions of tabular data instances (e.g., demographic and socioeconomic features) presented as text prompts.

Output: A risk score representing the predicted probability of the positive class, generated either via multiple-choice prompting (continuous token-probability space) or numeric prompting (discrete token space).

Scoring recipe

def compute_metrics(predictions, labels):
    n = len(predictions)
    # ECE (10 quantile bins)
    bins = np.array_split(predictions, 10)
    ece = sum(abs(np.mean(bin_preds) - np.mean(bin_labels)) for bin_preds, bin_labels in bins) / 10
    # Brier score
    brier = np.mean((np.array(predictions) - np.array(labels))**2)
    # AUC
    auc = roc_auc_score(labels, predictions)
    # Accuracy
    acc = np.mean(np.round(predictions) == labels)
    return ece, brier, auc, acc

Common pitfalls

• Numeric prompting forces discrete token outputs, causing tied risk scores and artificially lowering AUC compared to multiple-choice prompting.
• Base models tend to overestimate uncertainty (high variance, high uncertainty), while instruction-tuned models underestimate it (low variance, overconfident), leading to different calibration profiles.
• Calibration is evaluated using 10 quantile-based bins, which may behave differently on small or imbalanced test sets.

Evidence (verbatim from paper)

Figure A1 shows the change in calibration error (ECE) between using multiple-choice prompting and verbalized numeric prompting, on all five benchmark tasks. Instruction-tuned models (top rows) show ECE improvements on an overwhelming majority of model/task pairs, while base models (bottom rows) show less consistent results. However, using numeric prompting comes at a consistent cost of diminished predictive power (AUC) of the risk scores, shown in Figure A2.

Citation

@misc{cruz2024evaluating,
  title={Evaluating language models as risk scores},
  author={Cruz et al. (2024)},
  year={2024},
  note={arXiv:2407.14614}
}

• arXiv: 2407.14614