# Treeeval Eval

> TreeEval probes an LLM's ability to handle complex, adaptive reasoning through dynamically generated hierarchical questions. It evaluates how well a model's relative performance ranking aligns with established leaderboards like AlpacaEval2.0, while testing the framework's efficiency in distinguishing fine-grained capability differences without relying on static datasets. Use when the user wants to benchmark on TreeEval (Dynamic/Benchmark-Free), or asks about evaluating this task. Reports Spearman correlation ($ ho$).

- Skill: `qhjqhj00/treeeval-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/treeeval-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/treeeval-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/treeeval-eval

---


# treeeval-eval

> TreeEval: Benchmark-Free Evaluation of Large Language Models through Tree Planning — Li et al. (2024) (arXiv:2402.13125, 2024)

## What this evaluates

TreeEval probes an LLM's ability to handle complex, adaptive reasoning through dynamically generated hierarchical questions. It evaluates how well a model's relative performance ranking aligns with established leaderboards like AlpacaEval2.0, while testing the framework's efficiency in distinguishing fine-grained capability differences without relying on static datasets.

## Datasets

- **TreeEval (Dynamic/Benchmark-Free)** — total ?; splits: (unstated); repo https://github.com/Ashura5/TreeEval

## Metrics

- `Spearman correlation ($
ho$)` **(primary)** — range: other
  - Measures the rank correlation between the model rankings produced by TreeEval and the gold rankings from AlpacaEval2.0. Values range from -1 to 1, where 1 indicates perfect agreement in ranking order.
- `Kendall correlation ($	au$)` — range: other
  - Measures the ordinal association between TreeEval rankings and AlpacaEval2.0 rankings by comparing concordant and discordant pairs. Values range from -1 to 1.

## Input / output format

**Input**: A dynamically generated question or prompt produced by the examiner LLM (GPT-4) based on a hierarchical tree structure. Questions are tailored to specific topics and adapt in difficulty based on pairwise comparison progress.

**Output**: A natural language response to the generated question. The response is later compared pairwise against a reference model's response by the examiner LLM.

## Scoring recipe

```python
def compute_spearman_correlation(treeeval_scores, gold_scores):
    # treeeval_scores: list of aggregated TreeEval scores per model
    # gold_scores: list of AlpacaEval2.0 scores per model
    from scipy.stats import spearmanr
    rho, _ = spearmanr(treeeval_scores, gold_scores)
    return rho
```

## Common pitfalls

- TreeEval is benchmark-free and generates questions dynamically; results depend on the examiner's temperature (set to 1) and random seed, so exact reproducibility requires fixing these.
- Scores are relative pairwise comparisons against a fixed reference model (Mistral-7B-Instruct-v0.2), not absolute capability measures.
- The number of questions (#Q) varies per model pair; the reported average (#Q) is for efficiency comparison, not a fixed evaluation budget.

## Evidence (verbatim from paper)

> We report the correlation of rankings obtained through different methods with those from AlpacaEval2.0, using $	au$ for the Kendall correlation coefficient and $
ho$ for the Spearman correlation coefficient.

## Citation

```bibtex
@misc{li2024treeeval,
  title={TreeEval: Benchmark-Free Evaluation of Large Language Models through Tree Planning},
  author={Li et al. (2024)},
  year={2024},
  note={arXiv:2402.13125}
}
```

- arXiv: 2402.13125