Behavioral Prediction Eval

behavioral-prediction-eval

Decoding the Human Factor: High Fidelity Behavioral Prediction for Strategic Foresight — Yellin et al. (2026) (arXiv:2602.17222, 2026)

What this evaluates

Predicts individual strategic decisions by conditioning on structured psychometric trait profiles (e.g., Big Five personality traits). It probes a model's ability to map high-dimensional psychological embeddings to discrete behavioral outcomes in unseen situational contexts.

Datasets

• Strategic Scenario Dataset — total ?; splits: train (-1), test (-1)

Metrics

• balanced accuracy (primary) — range: [0, 1]
  • Computes the average recall across all classes, correcting for class imbalance by treating each class equally regardless of sample size.
• macro-F1 (primary) — range: [0, 1]
  • Calculates the unweighted mean of the F1 scores for each class, providing a harmonic mean of precision and recall per class.
• accuracy — range: [0, 1]
  • Standard classification accuracy: the proportion of correctly predicted labels out of the total number of predictions.

Input / output format

Input: Structured psychometric trait profiles (ranging from 5 to 74 traits) paired with a strategic scenario/question.

Output: Discrete class label corresponding to one of five multiple-choice behavioral outcomes.

Scoring recipe

def compute_metrics(y_true, y_pred, n_classes=5):
    acc = np.mean(y_true == y_pred)
    recalls = [np.mean(y_pred[y_true==c] == c) if np.sum(y_true==c) > 0 else 0 for c in range(n_classes)]
    bal_acc = np.mean(recalls)
    f1s = [f1_score([1 if y==c else 0 for y in y_true],
                    [1 if y==c else 0 for y in y_pred], average='binary') for c in range(n_classes)]
    macro_f1 = np.mean(f1s)
    return acc, bal_acc, macro_f1

Common pitfalls

• Class imbalance in response labels makes standard accuracy misleading; balanced accuracy and macro-F1 must be reported as primary metrics.
• Confidence intervals are computed via participant-level bootstrap resampling, not standard sample-level bootstrapping.
• Chance performance is approximately 0.20 due to the 5-way multiple-choice format, so scores near 0.20-0.30 are not necessarily poor.

Evidence (verbatim from paper)

Because the response labels are class-imbalanced, we report balanced accuracy and macro-F1 as primary metrics, as they better reflect performance across classes. We additionally report standard accuracy as a familiar reference measure.

Citation

@misc{yellin2026decoding,
  title={Decoding the Human Factor: High Fidelity Behavioral Prediction for Strategic Foresight},
  author={Yellin et al. (2026)},
  year={2026},
  note={arXiv:2602.17222}
}

• arXiv: 2602.17222