Profile My skills Saved Collections Edit profile Submit a skill

RAG Evaluation

Evaluate retrieval fixtures deterministically before provider-backed experiments.

prashantyr 2a40a92 2 files · 1.9 KB Updated
AI & ML

File contents

RAG Evaluation

Use when asked to measure a RAG retriever or inspect an evaluation fixture.

  1. Confirm the dataset is synthetic, public, or authorized; never send private text to a provider.
  2. Run python rag-evaluation/scripts/evaluate.py <fixture.json>.
  3. Report hit rate, failed query IDs, fixture size, and the exact command.
  4. Separate retrieval quality from answer quality. Do not claim semantic correctness from hit rate.
  5. If an Ollama experiment is requested, use the local endpoint in .env.example and preserve the deterministic baseline. For commercial models, require an explicit adapter and environment-only credential; never place a key in prompts, source files, or output.

Expected fixture keys are documents (id, text) and queries (id, query, expected_id). The safe script performs lexical ranking only, makes no network calls, and writes no files.

prashantyr/prashant-agent-skills/tree/main/rag-evaluation commit 2a40a9211f

Browse category

Frequently asked questions

npx skillmds@latest add prashantyr/rag-evaluation