Dora Eval

dora-eval

Domain-oriented RAG Assessment (DoRA): Synthetic Benchmarking for RAG-based Question Answering on Defense Documents — Doan et al. (2026) (arXiv:2604.17943, 2026)

What this evaluates

Evaluates RAG-based question answering systems on defense-domain documents, measuring both retrieval effectiveness and end-to-end QA performance including task success, faithfulness, and generation quality.

Datasets

• DoRA — total 6500; splits: train (4635), eval (515)

Metrics

• task-success (primary) — range: percent
  • Percentage of questions answered correctly according to reference answers or expert criteria.
• faithfulness — range: [0, 1]
  • RAGEval-based metric measuring completeness, hallucination, and irrelevance of generated answers relative to retrieved evidence.
• Tok. F1 — range: [0, 1]
  • F1 score computed over token-level overlap between predicted and reference answers.
• ROUGE-L — range: [0, 1]
  • Longest common subsequence recall/precision between predicted and reference answers.
• BLEU — range: [0, 1]
  • Bilingual evaluation understudy score measuring n-gram precision with brevity penalty.

Input / output format

Input: Question q concatenated with retrieved evidence context ctx(E) (or gold context E for oracle evaluation).

Output: Generated answer a.

Scoring recipe

def compute_metrics(predictions, references, contexts):
    token_f1 = [token_f1_score(p, r) for p, r in zip(predictions, references)]
    rouge_l = [rouge_l_score(p, r) for p, r in zip(predictions, references)]
    bleu = [bleu_score(p, r) for p, r in zip(predictions, references)]
    task_success = [is_correct(p, r) for p, r in zip(predictions, references)]
    faithfulness = [rag_faithfulness(p, c) for p, c in zip(predictions, contexts)]
    return {
        'token_f1': mean(token_f1),
        'rouge_l': mean(rouge_l),
        'bleu': mean(bleu),
        'task_success': mean(task_success),
        'faithfulness': mean(faithfulness)
    }

Common pitfalls

• Retrieval error often dominates performance, masking model generation quality; fixed retriever settings should be reported.
• Faithfulness metrics (completeness, hallucination, irrelevance) require careful alignment with retrieved evidence, not just reference answers.
• Synthetic data generation may introduce intent mismatch or contamination if seed documents overlap with training corpora.

Evidence (verbatim from paper)

Results in Table 2 show that under a fixed retrieval setting, general-purpose LLMs cluster closely on both task-success and faithfulness, suggesting that retrieval and grounding constraints dominate performance in this private, defense-domain corpus.

Citation

@misc{doan2026dora,
  title={Domain-oriented RAG Assessment (DoRA): Synthetic Benchmarking for RAG-based Question Answering on Defense Documents},
  author={Doan et al. (2026)},
  year={2026},
  note={arXiv:2604.17943}
}

• arXiv: 2604.17943