# RAG Evaluation

> Evaluate retrieval fixtures deterministically before provider-backed experiments.

- Skill: `prashantyr/rag-evaluation` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add prashantyr/rag-evaluation`
- Raw SKILL.md: https://api.skillmd.com/api/skills/prashantyr/rag-evaluation/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: prashantyr (https://skillmd.com/u/prashantyr)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/prashantyr/rag-evaluation

---


# RAG Evaluation

Use when asked to measure a RAG retriever or inspect an evaluation fixture.

1. Confirm the dataset is synthetic, public, or authorized; never send private text to a provider.
2. Run `python rag-evaluation/scripts/evaluate.py <fixture.json>`.
3. Report hit rate, failed query IDs, fixture size, and the exact command.
4. Separate retrieval quality from answer quality. Do not claim semantic correctness from hit rate.
5. If an Ollama experiment is requested, use the local endpoint in `.env.example` and preserve the
   deterministic baseline. For commercial models, require an explicit adapter and environment-only
   credential; never place a key in prompts, source files, or output.

Expected fixture keys are `documents` (`id`, `text`) and `queries` (`id`, `query`, `expected_id`).
The safe script performs lexical ranking only, makes no network calls, and writes no files.