Cfr Retrieval Eval

cfr-retrieval-eval

Knowledge Graph RAG: Agentic Crawling and Graph Construction in Enterprise Documents — Chakraborty et al. (2026) (arXiv:2604.14220, 2026)

What this evaluates

Evaluates an information retrieval system's ability to navigate complex, hierarchical, and temporally-varying regulatory documents. It probes the model's capacity to resolve dense cross-references and versioning conflicts to provide complete and accurate answers.

Datasets

• Code of Federal Regulations (CFR) — total ?; splits: test (20)

Metrics

• Accuracy (Correct/Complete Answers) (primary) — range: percent
  • Proportion of questions answered correctly and completely out of the total 20 curated regulatory questions. Answers are classified as Correct/Complete, Incomplete/Inaccurate, or Refusals.
• Overlap Coefficient — range: [0, 1]
  • Score = |Keywords_Answer ∩ Keywords_Source| / |Keywords_Answer|, where the numerator counts unique meaningful words shared between the generated answer and the gold source snippet, and the denominator is the total count of unique meaningful words in the answer.

Input / output format

Input: Complex regulatory questions requiring multi-hop traversal, temporal resolution, and hierarchical context (e.g., 'According to 561.2 from 2002, what constitutes an account...').

Output: Natural language text answers generated by the retrieval system.

Scoring recipe

def compute_accuracy(predictions, gold_questions):
    correct = sum(1 for p in predictions if is_correct_and_complete(p))
    return correct / len(gold_questions)

def compute_overlap_coefficient(answer, gold_source):
    ans_kw = set(extract_meaningful_words(answer))
    src_kw = set(extract_meaningful_words(gold_source))
    if not ans_kw: return 0.0
    return len(ans_kw & src_kw) / len(ans_kw)

Common pitfalls

• Treating explicit regulatory citations as mere text strings rather than navigable pointers, causing retrieval failures across cross-references.
• Failing to account for temporal precedence and document versioning, leading to the use of superseded or outdated clauses.
• Relying solely on semantic similarity, which misses logical relationships like exemptions, negations, or hierarchical dependencies.

Evidence (verbatim from paper)

Accuracy was measured using the Overlap Coefficient formula: Score = |Keywords_Answer ∩ Keywords_Source| / |Keywords_Answer| ... The Knowledge Graph approach demonstrated a 70% improvement in accuracy over the standard RAG baseline. While the RAG system failed to provide a complete or correct response for 70% of the queries—often due to incomplete context or retrieval "hallucinations"—the Knowledge Graph system successfully navigated the nested structures to provide exhaustive and precise answers in every instance.

Citation

@misc{chakraborty2026knowledgegraphrag,
  title={Knowledge Graph RAG: Agentic Crawling and Graph Construction in Enterprise Documents},
  author={Chakraborty et al. (2026)},
  year={2026},
  note={arXiv:2604.14220}
}

• arXiv: 2604.14220