RAG Tech Docs Eval

rag-tech-docs-eval

Observations on Building RAG Systems for Technical Documents — Soman et al. (2024) (arXiv:2404.00657, 2024)

What this evaluates

Evaluates how chunking strategies, embedding models, and retrieval thresholds affect RAG performance on technical IEEE documents. Probes the impact of sentence length, keyword position, and acronym handling on retrieval relevance and generator hallucination.

Datasets

• IEEE Wireless LAN MAC/PHY & Battery Glossary — total ?; splits: test (42)

Metrics

• qualitative observation (primary) — range: other
  • No formal numerical formula; evaluation relies on qualitative assessment of retrieval relevance and LLM generation quality (e.g., correctness, hallucination) across 42 domain-sourced queries.

Input / output format

Input: Technical document text (full paragraphs, split terms/definitions, or table-excluded documents) paired with a natural language question.

Output: Retrieved document chunks and an LLM-generated answer.

Scoring recipe

def evaluate_rag(doc_chunks, question, llm, embedding_model):
    q_emb = embedding_model.encode(question)
    c_embs = [embedding_model.encode(c) for c in doc_chunks]
    scores = [cosine(q_emb, e) for e in c_embs]
    top_chunks = retrieve_top_k(doc_chunks, scores, k=5)
    prompt = build_prompt(question, top_chunks)
    answer = llm.generate(prompt)
    # Qualitative assessment per paper protocol
    relevance = assess_relevance(top_chunks, question)
    hallucination = check_acronym_expansion(answer)
    return relevance, hallucination

Common pitfalls

• Assuming higher similarity scores guarantee correct answers (thresholding fails to correlate with correctness)
• Treating long sentences (>200 words) as standard chunks (distorts embeddings and causes spurious matches)
• Ignoring acronym handling in definitions (leads to hallucinated expansions in generator output)

Evidence (verbatim from paper)

We evaluate on multiple questions and report on selected questions to substantiate our observations.

Citation

@misc{soman2024observations,
  title={Observations on Building RAG Systems for Technical Documents},
  author={Soman et al. (2024)},
  year={2024},
  note={arXiv:2404.00657}
}

• arXiv: 2404.00657