Soccsci210 Eval

soccsci210-eval

Finetuning LLMs for Human Behavior Prediction in Social Science Experiments — Kolluri et al. (2025) (arXiv:2509.05830, 2025)

What this evaluates

This evaluation probes an LLM's ability to predict individual human responses in social science experiments and match the overall distribution of those responses. It measures both point-wise accuracy and distributional alignment across unseen studies, conditions, outcomes, and participant demographics.

Datasets

• SocSci210 — total 2900000; splits: train (-1), test (-1)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Proportion of correctly predicted individual responses out of the total number of predictions.
• Wasserstein distance (primary) — range: [0, 1]
  • Earth Mover's Distance between the predicted response distribution and the ground-truth human response distribution for each experimental condition or outcome.
• Relative change — range: percent
  • ((metric_method - metric_base) / metric_base) * 100%, with the sign assigned so that positive values indicate improvement over the baseline.
• Demographic parity difference — range: [0, 1]
  • Absolute gap between the highest and lowest average Wasserstein distances computed across demographic subgroups.

Input / output format

Input: Prompt containing the experimental stimulus/question plus all available demographic attributes of the target participant. Few-shot examples may be appended based on embedding similarity of stimuli.

Output: A single predicted response value matching the experiment's outcome scale (e.g., standardized numerical score or categorical choice).

Scoring recipe

def compute_metrics(predictions, gold, base_predictions=None):
    accuracy = sum(p == g for p, g in zip(predictions, gold)) / len(gold)
    wasserstein = wasserstein_1d(gold, predictions)
    if base_predictions is not None:
        base_acc = sum(p == g for p, g in zip(base_predictions, gold)) / len(gold)
        base_wass = wasserstein_1d(gold, base_predictions)
        rel_acc = ((accuracy - base_acc) / base_acc) * 100
        rel_wass = ((wasserstein - base_wass) / base_wass) * 100
    return {'accuracy': accuracy, 'wasserstein': wasserstein, 'rel_acc': rel_acc, 'rel_wass': rel_wass}

Common pitfalls

• Distributional alignment (Wasserstein distance) and individual accuracy can move in opposite directions; better distribution matching may lower point-wise accuracy.
• The 'Empirical Best' bound is not a model target but a data-variance upper bound derived from bootstrapping the original responses.
• Relative change percentages use absolute values in the denominator but assign signs based on improvement direction, which can cause confusion when interpreting negative values.

Evidence (verbatim from paper)

When computing the Wasserstein distance, we treat the responses in our dataset as the ground-truth distribution of human responses. Because individual responses naturally vary, this empirical sample may not perfectly capture the true distribution of outcomes under each condition. Our source experiments were highly powered to estimate a treatment effect of a certain size, not to robustly estimate the full distribution. To estimate an empirical upper bound on performance given this variance, we perform bootstrapping: we generate 100 resampled datasets (with replacement) from the original responses. For each bootstrapped dataset, we compute the Wasserstein distance between that resample and the full original sample, then average these distances across all 100 iterations.

Citation

@misc{kolluri2025soccsci,
  title={Finetuning LLMs for Human Behavior Prediction in Social Science Experiments},
  author={Kolluri et al. (2025)},
  year={2025},
  note={arXiv:2509.05830}
}

• arXiv: 2509.05830