LLM Latent Skill Eval

llm-latent-skill-eval

IQ Test for LLMs: An Evaluation Framework for Uncovering Core Skills in LLMs — Maimon et al. (2025) (arXiv:2507.20208, 2025)

What this evaluates

Evaluates LLMs across 44 existing tasks to uncover latent cognitive skills using psychometric factor analysis, rather than relying on aggregated benchmark scores. It probes whether models possess coherent, interpretable skill profiles across diverse domains like reading comprehension, mathematical reasoning, and ethical judgment.

Datasets

• SQuAD — total ?; splits: unspecified (-1)
• GSM8K — total ?; splits: unspecified (-1)
• GPQA — total ?; splits: unspecified (-1)
• TriviaQA — total ?; splits: unspecified (-1)
• XSum — total ?; splits: unspecified (-1)
• MNLI (textual entailment) — total ?; splits: unspecified (-1)
• Ethical/Social Judgment datasets — total ?; splits: unspecified (-1)

Metrics

• FA (primary) — range: continuous
  • A psychometric factor analysis method that decomposes the model-task performance matrix to infer latent constructs. It disentangles shared variance (core cognitive abilities) from task-specific noise, yielding interpretable skill dimensions.

Input / output format

Input: A performance matrix containing model scores across 44 heterogeneous evaluation tasks.

Output: Eight interpretable latent skill dimensions with associated proficiency scores for each evaluated model.

Scoring recipe

performance_matrix = load_model_task_scores(models, 44_tasks)
factors = perform_principal_axis_factoring(performance_matrix, n_factors=8)
skill_profiles = factors.transform(performance_matrix)
return skill_profiles

Common pitfalls

• Aggregating scores across heterogeneous tasks obscures true skill profiles and misleads model comparisons.
• Task redundancy is often ignored, causing over-representation of certain domains like textual entailment while under-representing ethical judgment.
• Inconsistent evaluation metrics across datasets make direct cross-task comparison difficult without normalization.

Evidence (verbatim from paper)

FA, a statistical method used to infer latent constructs from observed patterns in behavioral data, is a cornerstone of this tradition, and provides a principled way to interpret what is being measured, and to meaningfully evaluate individual differences.

Citation

@misc{maimon2025iqtestllms,
  title={IQ Test for LLMs: An Evaluation Framework for Uncovering Core Skills in LLMs},
  author={Maimon et al. (2025)},
  year={2025},
  note={arXiv:2507.20208}
}

• arXiv: 2507.20208