Persian RAG Eval

persian-rag-eval

Advancing Retrieval-Augmented Generation for Persian: Development of Language Models, Comprehensive Benchmarks, and Best Practices for Optimization — Bourbour et al. (2025) (arXiv:2501.04858, 2025)

What this evaluates

Evaluates retrieval-augmented generation (RAG) pipelines for Persian text across general, scientific, and formal domains. It probes the ability of sentence embeddings to retrieve relevant context and large language models to generate accurate, faithful, and relevant answers based on that context.

Datasets

• PQuad — total ?; splits: test (-1)
• Scientific-Specialized — total ?; splits: test (-1)
• Organizational Report — total ?; splits: test (-1)

Metrics

• Retrieval Accuracy (1st/2nd/3rd rank) — range: percent
  • Percentage of queries where the gold document appears in the top-1, top-2, or top-3 retrieved results, averaged across ranks.
• Context Recall (primary) — range: [0, 1]
  • LLM-as-judge metric measuring the extent to which the retrieved context covers all information needed to answer the query.
• Faithfulness — range: [0, 1]
  • LLM-as-judge metric evaluating whether the generated answer is fully grounded in and consistent with the retrieved context.
• Answer Relevancy — range: [0, 1]
  • LLM-as-judge metric scoring how directly the generated answer addresses the query given the context.
• Context Precision — range: [0, 1]
  • LLM-as-judge metric assessing whether the retrieved context chunks are precisely relevant to the query without extraneous information.

Input / output format

Input: Query string and retrieved context chunks (or document summaries) for retrieval; Query and context for generation.

Output: Ranked list of retrieved document chunks; Generated answer text.

Scoring recipe

def score_retrieval(retrieved, gold):
    hits = sum(1 for i, doc in enumerate(retrieved[:3]) if doc == gold)
    return hits / len(retrieved)

def score_ragas(query, context, answer):
    # LLM-as-judge scoring per RAGAS framework
    ctx_recall = judge(f'Does context fully answer query?')
    faith = judge(f'Is answer fully grounded in context?')
    ans_rel = judge(f'Is answer relevant to query?')
    ctx_prec = judge(f'Is context precisely relevant to query?')
    return ctx_recall, faith, ans_rel, ctx_prec

Common pitfalls

• Retrieval accuracy is evaluated at top-1, top-2, and top-3 ranks rather than standard recall@k or MRR.
• RAGAS metrics rely on LLM-as-a-judge scoring, which can be sensitive to prompt phrasing and model temperature settings.
• Chunk size and summary indexing drastically alter retrieval precision, particularly for formal/organizational texts, making direct comparisons across chunking strategies difficult without standardized settings.

Evidence (verbatim from paper)

The RAGAS framework provided a comprehensive evaluation of both retrieval and generation performance across all datasets. The results revealed significant variation in model performance, with larger models like LLaMA-3.1 (70B) consistently demonstrating the highest performance in terms of answer relevancy and context recall across all datasets.

Citation

@misc{bourbour2025advancing,
  title={Advancing Retrieval-Augmented Generation for Persian: Development of Language Models, Comprehensive Benchmarks, and Best Practices for Optimization},
  author={Bourbour et al. (2025)},
  year={2025},
  note={arXiv:2501.04858}
}

• arXiv: 2501.04858