Twin 2k 500 Eval

twin-2k-500-eval

Twin-2K-500: A dataset for building digital twins of over 2,000 people based on their answers to over 500 questions — Toubia et al. (2025) (arXiv:2505.17479, 2025)

What this evaluates

Evaluates the ability of LLMs to simulate individual-level human behavior across demographic, psychological, cognitive, economic, and behavioral economics domains. It measures test-retest accuracy and replication of known behavioral biases using a large-scale dataset of 2,058 U.S. individuals.

Datasets

• Twin-2K-500 — total 2058; splits: full (2058); repo https://github.com/tianyipeng-lab/Digital-Twin-Simulation

Metrics

• test-retest accuracy (primary) — range: [0, 1]
  • Calculated as the correlation or agreement rate between a participant's responses in Waves 1–3 and their repeated responses in Wave 4 for the same experimental conditions and questions.

Input / output format

Input: Individual-level answers to 500+ questions spanning demographics, personality, cognitive ability, economic preferences, and behavioral economics experiments, along with assigned experimental conditions.

Output: Simulated answers or behavioral predictions for the same questions and conditions, generated by an LLM digital twin conditioned on the individual's profile.

Scoring recipe

def compute_test_retest_accuracy(wave1_3_responses, wave4_responses):
    aligned = align_by_participant_and_question(wave1_3_responses, wave4_responses)
    accuracy = mean_agreement(aligned)
    return accuracy

Common pitfalls

• Evaluators must match Wave 4 responses to the exact same experimental conditions as Waves 1–3, as conditions were fixed per participant.
• The dataset mixes multiple response formats (Likert scales, binary choices, numerical estimates, open-ended text), requiring format-specific scoring rather than a single generic metric.
• Test-retest accuracy only applies to the 88 questions repeated in Wave 4, not the full 500+ question set.

Evidence (verbatim from paper)

Wave 4 repeated the within- and between-subject heuristics and biases experiments from the first three waves as well as the pricing study from wave 3 (88 questions in total). Participants were assigned to the exact same condition in wave 4 as they were in waves 1-3 for each of these experiments, providing us a clean measure of test-retest accuracy.

Citation

@misc{toubia2025twin2k500,
  title={Twin-2K-500: A dataset for building digital twins of over 2,000 people based on their answers to over 500 questions},
  author={Toubia et al. (2025)},
  year={2025},
  note={arXiv:2505.17479}
}

• arXiv: 2505.17479