QA Quality Eval

qa-quality-eval

RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture — Balaguer et al. (2024) (arXiv:2401.08406, 2024)

What this evaluates

Evaluates the quality of generated question-answer pairs in an agricultural domain context. It probes a model's ability to produce relevant, accurate, diverse, and fluent Q&A content under varying context conditions.

Datasets

• Agricultural Q&A dataset — total ?; splits: test (-1)

Metrics

• Relevance (primary) — range: [1, 5]
  • LLM-judged score on a 1-5 scale assessing how pertinent the generated answer is to the question.
• Coverage — range: [1, 5]
  • LLM-judged score on a 1-5 scale measuring how comprehensively the answer addresses the question and covers relevant source text sections.
• Diversity — range: other
  • Computed using Word Mover's Distance (WMD) or KL divergence to measure semantic dissimilarity between generated questions and source text; lower scores indicate higher diversity.
• Overlap — range: other
  • Computed using KL divergence or WMD to measure semantic similarity between generated questions and source text; higher scores indicate more overlap.
• Fluency — range: [1, 5]
  • LLM-judged score on a 1-5 scale evaluating the grammatical correctness and natural flow of the generated answer.
• Correctness — range: [1, 5]
  • LLM-judged categorical score (e.g., Correct, Partially Correct, Incorrect) mapped to a 1-5 scale based on factual alignment with the reference answer.
• Succinctness — range: [1, 5]
  • LLM-judged score on a 1-5 scale penalizing verbose or overly detailed responses that go beyond the scope of the question.

Input / output format

Input: Question, generated answer, reference answer, and optional context (no context, location/state context, or external geographic context).

Output: A numerical score from 1 to 5 for each metric, accompanied by a textual explanation justifying the score.

Scoring recipe

def evaluate_qa(question, answer, reference, context, metric):
    if metric in ['Overlap', 'Diversity']:
        return compute_wmd_or_kl(question, context)
    else:
        prompt = f'Score the {metric} of this answer (1-5): Q: {question} A: {answer} Ref: {reference}'
        return llm_judge_score(prompt, scale=5)

# Aggregate across all instances
final_score = mean([evaluate_qa(q, a, r, c, m) for q, a, r, c in dataset])

Common pitfalls

• LLM judges tend to penalize concise answers if they lack elaboration, as seen in the Succinctness metric where verbose but relevant answers score lower.
• Diversity and Overlap require specific distance metrics (WMD/KL divergence) rather than simple LLM scoring, which can lead to implementation inconsistencies if not strictly followed.
• Context setup variations (no context vs. external context) drastically alter prompt length and model behavior, making direct metric comparisons across setups sensitive to formatting.

Evidence (verbatim from paper)

We used several metrics (defined on Section 4.1) to assess their quality, including Relevance, Global Relevance, Coverage, Overlap, Diversity, Details, and Fluency. The LLMs assessed the Q&A pairs on various metrics, scoring each on a scale from 1 to 5. For certain metrics such as Overlap and Diversity, we incorporated intricate methods like Kullback-Leibler (KL) divergence and Word Mover's Distance (WMD) to measure the semantic similarity between the source text and the questions generated.

Citation

@misc{balaguer2024ragvsfinetuning,
  title={RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture},
  author={Balaguer et al. (2024)},
  year={2024},
  note={arXiv:2401.08406}
}

• arXiv: 2401.08406