Leanrag Eval

leanrag-eval

LeanRAG: Knowledge-Graph-Based Generation with Semantic Aggregation and Hierarchical Retrieval — Zhang et al. (arXiv:2508.10391, 2025)

What this evaluates

Evaluates the quality of answers generated by RAG systems across specialized domains. It probes the model's ability to retrieve relevant information, synthesize comprehensive responses, and maintain diversity and practical utility. Additionally, it measures retrieval efficiency and the impact of structural knowledge on generation.

Datasets

• UltraDomain — total ?; splits: test (-1)

Metrics

• Comprehensiveness (primary) — range: [1, 10]
  • LLM-as-judge scores the answer's thoroughness in addressing the query on a 1-10 scale.
• Empowerment — range: [1, 10]
  • LLM-as-judge evaluates the answer's practical utility and actionability on a 1-10 scale.
• Diversity — range: [1, 10]
  • LLM-as-judge assesses the breadth of information and perspectives presented on a 1-10 scale.
• Overall — range: [1, 10]
  • LLM-as-judge provides a holistic quality score considering all dimensions on a 1-10 scale.
• Win Rate — range: percent
  • Pairwise comparison where an LLM judge selects the better answer between two methods; reported as percentage.
• Retrieval Token Count — range: other
  • Total number of tokens in the retrieved context used for generation; lower indicates less redundancy.

Input / output format

Input: User query paired with retrieved context (text chunks or structured graph entities with descriptions).

Output: Generated natural language answer text.

Scoring recipe

def compute_llm_judge_score(predictions, queries, prompt_template):
    all_scores = []
    for pred, query in zip(predictions, queries):
        scores = []
        for _ in range(5):  # Each query scored 5 times per paper
            score = llm_judge(prompt_template.format(query=query, answer=pred))
            scores.append(score)
        all_scores.append(sum(scores) / len(scores))
    return sum(all_scores) / len(all_scores)

Common pitfalls

• LLM-as-judge scores can vary significantly based on prompt phrasing, model version, and temperature settings.
• Token count measures syntactic redundancy, not semantic overlap, which may misrepresent actual information density.
• Pairwise win rates are sensitive to the choice of the opposing baseline model and may not reflect absolute quality.

Evidence (verbatim from paper)

To provide a multi-faceted and in-depth analysis of system performance, we evaluate the generated answers along four crucial dimensions follows the prior work (Huang et al. [2025a]): Comprehensiveness: Measures how thoroughly the answer addresses the user’s query. Empowerment: Evaluates the answer’s practical utility and its ability to provide actionable information. Diversity: Assesses the breadth of information and perspectives presented in the answer. Overall: Provides a single, holistic quality score to measure how answer perform overall, considering comprehensiveness, empowerment, diversity, and any other relevant factors. Following recent best practices in automated evaluation, we employ powerful LLMs as judges to score the outputs of all methods on the 1 to 10 scale defined by our metrics.

Citation

@misc{zhang2025leanrag,
  title={LeanRAG: Knowledge-Graph-Based Generation with Semantic Aggregation and Hierarchical Retrieval},
  author={Zhang et al.},
  year={2025},
  note={arXiv:2508.10391}
}

• arXiv: 2508.10391