RAG Implementation

RAG Implementation Workflow

Overview

This skill curates the upstream rag-implementation workflow into an execution-focused guide for designing, reviewing, and troubleshooting Retrieval-Augmented Generation systems.

Use it when the task is not just “add a vector database,” but to make concrete decisions about:

• whether RAG is the right solution at all
• whether managed file search is sufficient or custom indexing is required
• how to chunk, enrich, and index documents safely
• how to choose dense, lexical, hybrid, and filtered retrieval patterns
• how to evaluate retrieval quality separately from generation quality
• how to debug failures such as missed passages, stale citations, duplicate chunks, and weak grounding

Preserve the upstream intent: this remains a practical implementation workflow covering embedding selection, vector storage, chunking, and retrieval optimization. The enhancement adds stronger activation boundaries, clearer quality gates, and more operational troubleshooting.

When to Use This Skill

Use this skill when one or more of these are true:

• The system must answer from external documents rather than model memory.
• The source corpus changes often enough that prompt-only approaches become stale.
• The user requires citations, provenance, or document-grounded answers.
• The corpus is large enough that manual prompt stuffing is not realistic.
• Retrieval quality, filtering, freshness, or tenant isolation are part of the implementation scope.
• The team needs to compare managed retrieval with a custom vector pipeline.

Do not default to this skill when:

• The knowledge is small, static, and can fit directly in a prompt.
• The task is mainly style adaptation or behavior shaping rather than external knowledge access.
• There is no searchable corpus yet.
• Latency or complexity budgets do not allow retrieval, indexing, and evaluation overhead.
• A deterministic search/index system already exists and only answer formatting is needed.

Fast routing

Operating Table

For compact decision support, use:

• references/rag-decision-matrix.md
• examples/rag-evaluation-example.md

Workflow

Phase 1: Requirements and activation check

Inputs

• user task
• target corpus or planned corpus
• answer quality expectations
• latency, cost, and compliance constraints

Actions

• Confirm whether RAG is actually needed.
• Identify whether freshness, provenance, or grounded answers are required.
• Define what the system must return: answer only, answer plus citations, or structured diagnostics.
• Decide whether managed file search can satisfy the use case before proposing a custom stack.

Outputs

• a clear activation decision
• initial architecture direction: prompt-only, managed retrieval, or custom RAG
• explicit success criteria

Exit criteria

• The team can state why RAG is required.
• The team knows what counts as a correct answer and what evidence must accompany it.

Phase 2: Corpus preparation

Inputs

• source documents
• update cadence
• ownership and access rules

Actions

• inventory source types: policies, manuals, tickets, code, FAQs, tables, transcripts, product docs
• remove duplicate or superseded content where possible
• normalize encoding and extraction quality
• assign metadata needed later for filtering and auditability

Recommended metadata per chunk

• source URI or document id
• title or section label
• version or effective date
• ingestion timestamp
• language
• tenant, team, or ACL scope where applicable
• document type
• parent section lineage

Outputs

• clean corpus ready for chunking and indexing
• metadata schema for retrieval and security controls

Exit criteria

• Operators can trace any chunk back to its source.
• Access-control and freshness metadata exist before indexing.

Phase 3: Chunking and enrichment

Inputs

• normalized documents
• document-type inventory

Actions

• chunk by semantic boundaries first: headings, sections, paragraphs, FAQ items, code units, table neighborhoods
• use overlap only where it preserves context across boundaries
• keep surrounding structural cues that help retrieval and citation
• test different chunking strategies on real questions instead of adopting universal token defaults

Document-specific heuristics

• Policies / legal / compliance text: preserve clause and section boundaries; attach effective date and policy id.
• Technical docs: keep headings, version tags, API names, and nearby examples together.
• Code or config docs: chunk by function, class, command, or config block; avoid splitting syntax from explanation.
• FAQs / support articles: one question-answer pair per chunk is often clearer than broad paragraph chunks.
• Tables: keep the caption, header row meaning, and nearby explanatory text with extracted values where possible.

Outputs

• chunked corpus with structural metadata

Exit criteria

• A reviewer can inspect a chunk and still understand what document section it came from.
• Chunk boundaries do not destroy the meaning needed for retrieval.

Phase 4: Embedding and index design

Inputs

• chunked corpus
• query patterns
• scale and latency targets

Actions

• choose embedding approach appropriate to corpus and query language
• choose storage/index approach based on workload, not vendor preference
• decide whether metadata filtering, hybrid retrieval, reranking, or database-native indexing are required
• document re-embedding and reindex triggers before launch

Common decision factors

• corpus size and growth rate
• latency target
• lexical search importance for identifiers and exact terms
• need for metadata filters and ACL enforcement
• operational tolerance for running a separate search service
• observability and debugging needs

Reindex or re-embed when

• the chunk schema changes
• key metadata fields are added or corrected
• a major document refresh lands
• the embedding model changes
• retrieval quality regresses on a stable test set

Outputs

• documented embedding and index plan

Exit criteria

• The team can explain why this storage/index path fits the workload.
• Reindex triggers are known in advance, not discovered during incidents.

Phase 5: Retrieval design

Inputs

• indexed corpus
• representative user queries

Actions

• start with the simplest retrieval path that matches the corpus
• test dense retrieval, lexical retrieval, or hybrid retrieval against real queries
• apply metadata filters for tenant, freshness, product/version, language, or document scope
• tune top-k only after inspecting what is being returned
• consider reranking when initial recall is acceptable but final ordering is weak

Rules of thumb

• Use dense retrieval for semantic similarity and paraphrased questions.
• Use lexical signals when exact identifiers, product names, versions, or policy labels matter.
• Use hybrid retrieval when either semantic-only or keyword-only search misses relevant evidence.
• Use metadata filters as both quality controls and security controls.

Outputs

• retrieval policy for query handling
• logging requirements for top results, scores, filters, and citations

Exit criteria

• The team can inspect retrieved chunks and explain why they were selected.
• The system can narrow retrieval scope safely using metadata.

Phase 6: Grounded answer generation

Inputs

• retrieved chunks
• answer policy

Actions

• instruct the model to answer from retrieved evidence when grounding is required
• require citations or source references when the use case depends on provenance
• define abstention behavior when retrieval is weak, missing, or contradictory
• prefer structured outputs for diagnostics, eval runs, or review workflows

Minimum answer policy

• answer only from retrieved evidence when the task requires grounding
• cite the supporting source or section when feasible
• state uncertainty or ask a follow-up when the evidence is insufficient
• do not silently fill missing facts from general model knowledge in a supposedly grounded workflow

Outputs

• grounded answer contract
• operator-visible diagnostic format if needed

Exit criteria

• The answer behavior makes grounding failures visible rather than hiding them behind fluent prose.

Phase 7: Offline evaluation

Inputs

• gold query set
• expected documents or passages
• generated answers with citations

Actions

• evaluate retrieval quality separately from answer quality
• record whether the right document or passage appeared in the retrieved set
• verify citation correctness and unsupported claims
• group failures into buckets before changing prompts or models

Useful evaluation dimensions

• retrieval hit rate or recall proxy on expected documents/passages
• citation correctness
• groundedness or unsupported-claim rate
• answer usefulness to the user task
• abstention quality when evidence is weak

Outputs

• failure buckets tied to retrieval, chunking, metadata, ranking, or generation behavior

Exit criteria

• The team knows whether the main problem is ingestion/retrieval or answer generation.
• Prompt tuning is not used to hide indexing defects.

See examples/rag-evaluation-example.md for a worked mini-evaluation.

Phase 8: Production monitoring and maintenance

Inputs

• live queries
• retrieval logs
• corpus refresh events

Actions

• monitor retrieval misses, stale citations, empty-result rates, and filter behavior
• audit tenant or ACL scoping regularly
• track corpus drift and reindex triggers
• review examples where users report “hallucination” to confirm whether the real issue was retrieval failure

Outputs

• maintenance plan for refresh, reindex, and incident review

Exit criteria

• The team can detect degradation caused by corpus changes, not just model changes.

Troubleshooting

1. Relevant document exists but is not retrieved

Likely causes

• chunk boundaries split the answer from its heading or context
• semantic-only retrieval misses exact identifiers
• metadata filters are too narrow
• stale index or incomplete ingestion

Checks

• inspect top-k retrieved chunks and their metadata
• test the same query with and without filters
• test dense-only versus hybrid retrieval
• verify the document was actually indexed in the expected version

Fixes

• rework chunking to preserve semantic units
• add lexical or hybrid retrieval
• correct filters or metadata
• reindex the missing or updated content

2. Answer cites the wrong section or wrong version

Likely causes

• overlapping chunks produce near-duplicate candidates
• superseded content remains searchable
• ranking favors semantically similar but outdated text

Checks

• compare cited chunk metadata with effective date and version
• inspect for duplicate or superseded documents in the index
• review whether freshness metadata exists and is used

Fixes

• deduplicate or retire old content
• filter by version/effective date where appropriate
• store stronger provenance metadata and expose it in the answer

3. Answers are repetitive or contain duplicated evidence

Likely causes

• overlap is too high
• many near-identical chunks from the same source dominate retrieval
• top-k is too large for the query

Checks

• inspect neighboring retrieved chunks for near-duplicates
• compare answer quality at lower top-k values

Fixes

• reduce unnecessary overlap
• deduplicate chunk candidates before generation
• tune top-k based on query class, not a universal default

4. Dense retrieval fails on SKUs, codes, policy IDs, or version numbers

Likely causes

• lexical specificity matters more than semantic similarity
• identifiers were normalized or lost during ingestion

Checks

• run exact-term tests against representative identifier queries
• confirm identifiers remain present in chunk text and metadata

Fixes

• add lexical or hybrid retrieval
• preserve identifiers in chunk text and metadata
• add filters for product, version, or document type

5. Users report hallucinations, but the real issue is bad retrieval

Likely causes

• answer generation is blamed before retrieval logs are inspected
• weak evidence is still passed to the model as if it were sufficient

Checks

• review retrieved chunks before reviewing prompt wording
• verify whether cited evidence truly supports the answer
• test abstention behavior on weak-retrieval cases

Fixes

• enforce grounded-answer policy
• require citations in evaluation runs
• adjust retrieval, chunking, or ranking before changing prompts

6. Cross-tenant or unauthorized content appears in results

Likely causes

• missing ACL metadata
• retrieval performed without required filters
• shared index without proper scope enforcement

Checks

• inspect metadata fields on returned chunks
• confirm filters are applied server-side where required
• review ingestion pipeline for missing tenant or ACL attributes

Fixes

• add mandatory scope metadata to every chunk
• require retrieval filters for tenant/access boundaries
• reindex content after metadata correction

7. Retrieved content is stale or superseded

Likely causes

• corpus refresh does not trigger reindexing
• old and new versions coexist without ranking or filtering rules

Checks

• compare source freshness with index freshness
• inspect effective dates and ingestion timestamps

Fixes

• define explicit reindex triggers
• filter or rank by freshness where the use case requires it
• retire superseded content from searchable scope

8. Retrieved passages contain prompt injection or untrusted instructions

Likely causes

• the system treats retrieved text as trusted instructions instead of untrusted evidence
• unreviewed external content is indexed without policy boundaries

Checks

• inspect whether the prompt clearly separates system policy from retrieved content
• review the provenance and trust class of indexed sources

Fixes

• treat retrieved text as data, not instructions
• limit indexing of untrusted sources or isolate them by policy
• require the model to ground answers in evidence rather than obey document-embedded commands

Additional Resources

• references/rag-decision-matrix.md — compact architecture and retrieval decision matrix for implementation planning
• examples/rag-evaluation-example.md — worked example that separates retrieval defects from generation defects

Primary guidance to verify before final implementation decisions:

• OpenAI Embeddings guide: https://platform.openai.com/docs/guides/embeddings
• OpenAI Retrieval guide: https://platform.openai.com/docs/guides/retrieval
• OpenAI File Search guide: https://platform.openai.com/docs/guides/tools-file-search
• OpenAI structured output guidance: https://platform.openai.com/docs/guides/text?api-mode=responses

If exact documentation URLs move, re-check current official docs before merge or handoff.

Related Skills

Use a different or adjacent skill when the task shifts to:

• prompt engineering without external retrieval
• search relevance tuning for a non-LLM application
• evaluation framework design beyond a lightweight gold-set review
• agent orchestration where retrieval is one tool among many
• data governance or redaction workflows before indexing sensitive corpora