Treeeval Eval

treeeval-eval

TreeEval: Benchmark-Free Evaluation of Large Language Models through Tree Planning — Li et al. (2024) (arXiv:2402.13125, 2024)

What this evaluates

TreeEval probes an LLM's ability to handle complex, adaptive reasoning through dynamically generated hierarchical questions. It evaluates how well a model's relative performance ranking aligns with established leaderboards like AlpacaEval2.0, while testing the framework's efficiency in distinguishing fine-grained capability differences without relying on static datasets.

Datasets

• TreeEval (Dynamic/Benchmark-Free) — total ?; splits: (unstated); repo https://github.com/Ashura5/TreeEval

Metrics

• Spearman correlation ($ ho$) (primary) — range: other
  • Measures the rank correlation between the model rankings produced by TreeEval and the gold rankings from AlpacaEval2.0. Values range from -1 to 1, where 1 indicates perfect agreement in ranking order.
• Kendall correlation ($ au$) — range: other
  • Measures the ordinal association between TreeEval rankings and AlpacaEval2.0 rankings by comparing concordant and discordant pairs. Values range from -1 to 1.

Input / output format

Input: A dynamically generated question or prompt produced by the examiner LLM (GPT-4) based on a hierarchical tree structure. Questions are tailored to specific topics and adapt in difficulty based on pairwise comparison progress.

Output: A natural language response to the generated question. The response is later compared pairwise against a reference model's response by the examiner LLM.

Scoring recipe

def compute_spearman_correlation(treeeval_scores, gold_scores):
    # treeeval_scores: list of aggregated TreeEval scores per model
    # gold_scores: list of AlpacaEval2.0 scores per model
    from scipy.stats import spearmanr
    rho, _ = spearmanr(treeeval_scores, gold_scores)
    return rho

Common pitfalls

• TreeEval is benchmark-free and generates questions dynamically; results depend on the examiner's temperature (set to 1) and random seed, so exact reproducibility requires fixing these.
• Scores are relative pairwise comparisons against a fixed reference model (Mistral-7B-Instruct-v0.2), not absolute capability measures.
• The number of questions (#Q) varies per model pair; the reported average (#Q) is for efficiency comparison, not a fixed evaluation budget.

Evidence (verbatim from paper)

We report the correlation of rankings obtained through different methods with those from AlpacaEval2.0, using $ au$ for the Kendall correlation coefficient and $ ho$ for the Spearman correlation coefficient.

Citation

@misc{li2024treeeval,
  title={TreeEval: Benchmark-Free Evaluation of Large Language Models through Tree Planning},
  author={Li et al. (2024)},
  year={2024},
  note={arXiv:2402.13125}
}

• arXiv: 2402.13125