RAG Evaluation

Evaluate retrieval fixtures deterministically before provider-backed experiments.

prashantyr Updated

File contents

RAG Evaluation

Use when asked to measure a RAG retriever or inspect an evaluation fixture.

  1. Confirm the dataset is synthetic, public, or authorized; never send private text to a provider.
  2. Run python rag-evaluation/scripts/evaluate.py <fixture.json>.
  3. Report hit rate, failed query IDs, fixture size, and the exact command.
  4. Separate retrieval quality from answer quality. Do not claim semantic correctness from hit rate.
  5. If an Ollama experiment is requested, use the local endpoint in .env.example and preserve the deterministic baseline. For commercial models, require an explicit adapter and environment-only credential; never place a key in prompts, source files, or output.

Expected fixture keys are documents (id, text) and queries (id, query, expected_id). The safe script performs lexical ranking only, makes no network calls, and writes no files.

prashantyr/prashant-agent-skills commit 2a40a9211f

Frequently asked questions

npx get-skill add prashantyr/rag-evaluation