Hirag Eval

hirag-eval

Retrieval-Augmented Generation with Hierarchical Knowledge — Haoyu Huang et al. (2025) (arXiv:2503.10150, 2025)

What this evaluates

Evaluates the answer quality of Retrieval-Augmented Generation (RAG) systems across specialized domains. It measures how well generated responses address queries in terms of comprehensiveness, empowerment, diversity, and overall performance using pairwise LLM-as-a-judge comparisons.

Datasets

• UltraDomain — total ?; splits: test (-1)

Metrics

• win rate (primary) — range: percent
  • Percentage of instances where a method's answer is judged superior to a baseline's answer by an LLM judge across four dimensions: Comprehensiveness, Empowerment, Diversity, and Overall.

Input / output format

Input: Query, answer from method A, answer from method B (order alternated for fairness)

Output: LLM judge selects the superior answer per dimension (or declares a tie)

Scoring recipe

wins = 0
total = 0
for query, ans_A, ans_B in dataset:
    judge_A = llm_judge(ans_A, ans_B, dimensions)
    judge_B = llm_judge(ans_B, ans_A, dimensions) # order alternated
    if judge_A > judge_B: wins += 1
    total += 1
win_rate = (wins / total) * 100

Common pitfalls

• LLM-as-a-judge can exhibit position bias, mitigated here by alternating answer order.
• Dimensions like 'Empowerment' and 'Diversity' are subjective and depend heavily on the prompt instructions given to the judge LLM.
• Win rates are calculated pairwise against each baseline separately, not as a single leaderboard ranking.

Evidence (verbatim from paper)

We used the win rate to compare different methods, which indicates the percentage of instances that a method generates higher-quality answers compared to another method as judged by the LLM. We utilized GPT-4o as the evaluation model to judge which method generates a superior answer for each query for the following four dimensions: (1) Comprehensiveness: how thoroughly does the answer address the question, covering all relevant aspects and details? (2) Empowerment: how effectively does the answer provide actionable insights or solutions that empower the user to take meaningful steps? (3) Diversity: how well does the answer incorporate a variety of perspectives, approaches, or solutions to the problem? (4) Overall: how does the answer perform overall, considering comprehensiveness, empowerment, diversity, and any other relevant factors?

Citation

@misc{huang2025hirag,
  title={Retrieval-Augmented Generation with Hierarchical Knowledge},
  author={Haoyu Huang et al. (2025)},
  year={2025},
  note={arXiv:2503.10150}
}

• arXiv: 2503.10150