RAG Data Curator

RAG Data Curator

Purpose

Turn raw Markdown or plain-text documents into reviewable chunks that are ready for embedding and vector-store ingestion. Treat curation as a data-quality workflow: inspect before transforming, preserve semantic structure, record provenance, and report measured results separately from recommended settings.

Tools determine what an agent can do; this skill defines how the agent should do document curation consistently.

When to use

Use this skill to:

• prepare .md, .markdown, or .txt corpora for a RAG pipeline;
• replace blind fixed-window splitting with structure-aware chunking;
• generate chunk-level provenance and quality evidence;
• inspect an existing chunk dataset for fragmentation, duplication, or metadata gaps.

Do not treat this workflow as a retrieval benchmark. Chunk quality signals cannot establish answer quality without downstream evaluation.

Goals

• Preserve headings, paragraphs, sentences, fenced code, and useful blockquotes where possible.
• Keep each accepted chunk within the configured token maximum.
• Carry only metadata declared by the source or derived through documented deterministic rules.
• Produce stable identifiers, normalized text hashes, and a measured quality report.
• Leave a clear audit trail from each chunk to its source document and section.

Workflow

Follow these stages in order:

1. Inspect. Count headings, paragraphs, sentences, fenced code blocks, blockquotes, exact repeated blocks, known boilerplate, and malformed code fences. Decide whether code and quotes have retrieval value before dropping them.
2. Normalize. Normalize line endings and prose whitespace. Remove only conservative boilerplate and exact repeated substantive blocks. Preserve headings and paragraph boundaries. Keep code and blockquotes by default.
3. Chunk. Split at section, heading, paragraph, and sentence boundaries in that order. Use the token limit only as a fallback for a single oversized semantic unit.
4. Add overlap. For adjacent chunks in the same section, copy up to the last two complete sentences from the previous chunk. Aim for the configured overlap target and skip overlap if no full sentence fits.
5. Enrich metadata. Record provenance, title, section, index, token count, tags, and keywords. Add optional domain fields only when they appear in source frontmatter.
6. Validate. Check empty, short, oversized, duplicated, fallback-split, heading-only, metadata-deficient, and excessive-overlap chunks. Reject empty, oversized, and exact duplicate chunks; report nonfatal findings.
7. Deliver. Write chunk JSON plus a Markdown quality report, and state the tokenizer and run configuration used for every measured size.

Before changing the rules for a corpus, read references/chunking-strategy.md. Before mapping source fields, read references/metadata-schema.md. Use references/quality-checklist.md when reviewing the output or designing a downstream gate.

Chunking rules

Start with these recommended defaults:

• target chunk size: 400 tokens;
• maximum chunk size: 600 tokens;
• minimum chunk size: 80 tokens;
• overlap target: 60 tokens.

These values are tunable starting points, not industry standards. Prefer ending near the target at a complete semantic boundary. Never exceed the maximum; if a single sentence or code block exceeds it, use a tokenizer-based fallback split and report that fallback. Merge an undersized prose fragment with a neighbor in the same section when the combined chunk remains within the maximum. A naturally complete short section may remain independent.

Metadata rules

Every chunk must include:

• source, filename, document_title, and section_title;
• chunk_index and measured token_count;
• tags and keywords, which may be empty arrays when the source provides no reliable values.

The bundled implementation derives document_title from declared title, then the first H1, then the filename. It derives section_title from the nearest heading, and uses declared tags/keywords plus document and section titles. It copies author, category, and dynasty only when those fields exist in frontmatter. Do not infer identities, dates, categories, or domain attributes from unstated assumptions.

Quality checks

Apply the acceptance rules and interpretations in references/quality-checklist.md. A clean report means the configured structural checks passed; it does not prove good retrieval, ranking, or generation performance.

Output format

Emit one JSON object per accepted chunk with this shape:

{
  "chunk_id": "stable-content-derived-id",
  "content": "chunk text",
  "metadata": {
    "source": "relative/path.md",
    "filename": "path.md",
    "document_title": "Document title",
    "section_title": "Section title",
    "chunk_index": 0,
    "token_count": 386,
    "overlap_token_count": 0,
    "tags": [],
    "keywords": []
  },
  "token_count": 386,
  "normalized_text_hash": "sha256:..."
}

The quality report must include document count, accepted chunk count, measured average/minimum/maximum size, duplicate count and rate, rejected count, metadata completeness, inspection counts, configuration, and individual findings.

Run the bundled implementation

Install and curate a corpus:

python -m pip install -e .
python scripts/curate.py docs/ \
  --output build/chunks.json \
  --report build/quality-report.md \
  --source-root docs/

Review both outputs before embedding. Adjust parameters only with a corpus-specific reason, then record the configuration and compare downstream retrieval behavior.

Failure and fallback behavior

• If no supported documents are found, stop and report the paths checked.
• If the configured tokenizer is unavailable, stop and install/configure it; do not silently substitute character counts.
• If a semantic unit exceeds the maximum, apply the measured token fallback, flag broken_sentence_fallback, and keep the source traceable.
• If source metadata is absent or ambiguous, leave optional values absent and required list fields empty.
• If parsing reveals an unclosed code fence or widespread noise, include it in inspection results and request human review before ingestion.
• If clean chunking is impossible without losing important structure, preserve the source and report the limitation rather than hiding it.

Important constraints

• Analyze structure before splitting.
• Prefer semantic boundaries; use token limits as a safety bound.
• Never fabricate metadata, benchmarks, user counts, performance gains, or downstream retrieval claims.
• Distinguish configured recommendations from measurements observed in the current run.
• Preserve source text fidelity except for declared normalization and overlap.
• Treat successful preprocessing and successful RAG retrieval as separate claims requiring separate evidence.