Qdrant Manager

Qdrant Manager

Agent: AIEngineer

Manage Qdrant Cloud collections for the Physical AI textbook RAG system. This skill handles vectorization of textbook chapters, metadata tagging, and collection management for efficient retrieval-augmented generation.

Quick Setup

# Set environment variables
export QDRANT_URL="https://xxx.qdrant.io"
export QDRANT_API_KEY="your-api-key"
export OPENAI_API_KEY="sk-..."

# Initialize collection
.claude/skills/qdrant-manager/scripts/setup.sh --init

# Vectorize all chapters
.claude/skills/qdrant-manager/scripts/setup.sh --vectorize docs/

# Query the collection
.claude/skills/qdrant-manager/scripts/setup.sh --query "What is ROS 2?"

Command Options

What It Does

1. Collection Management

Creates and manages Qdrant collections with optimized settings for textbook content:

# Collection schema
{
    "vectors": {
        "size": 1536,  # text-embedding-3-small
        "distance": "Cosine"
    },
    "on_disk_payload": True  # Optimize for large documents
}

2. Chapter Vectorization

Processes markdown files with intelligent chunking:

• Chunk Strategy: Split by headers (##, ###) for semantic coherence
• Overlap: 100 tokens overlap between chunks
• Max Chunk Size: 500 tokens per chunk
• Metadata Extraction: Title, chapter, section, language, keywords

3. Metadata Schema

Each vector point includes rich metadata for filtered retrieval:

{
    "id": "uuid-v4",
    "vector": [0.1, 0.2, ...],
    "payload": {
        "content": "The actual text content...",
        "source": "docs/chapter-1/index.md",
        "chapter": 1,
        "section": "Introduction",
        "title": "Introduction to Physical AI & ROS 2",
        "language": "en",
        "locale": "en-US",
        "keywords": ["ros2", "physical-ai", "robotics"],
        "word_count": 245,
        "created_at": "2026-01-02T00:00:00Z",
        "chunk_index": 0,
        "total_chunks": 5
    }
}

4. RAG Query Support

Semantic search with metadata filtering:

# Example: Query with language filter
results = client.search(
    collection_name="textbook_chapters",
    query_vector=embed("What is ROS 2?"),
    query_filter={
        "must": [
            {"key": "language", "match": {"value": "en"}}
        ]
    },
    limit=5
)

Bundled Resources

1. Python Dependencies

File: requirements.txt

qdrant-client>=1.7.0
openai>=1.0.0
tiktoken>=0.5.0
python-dotenv>=1.0.0
rich>=13.0.0

2. Collection Configuration

File: assets/collection_config.json

{
    "vectors": {
        "size": 1536,
        "distance": "Cosine"
    },
    "optimizers_config": {
        "indexing_threshold": 20000
    },
    "on_disk_payload": true,
    "replication_factor": 1
}

3. Chunking Configuration

File: assets/chunking_config.json

{
    "chunk_size": 500,
    "chunk_overlap": 100,
    "separators": ["## ", "### ", "\n\n", "\n"],
    "length_function": "tiktoken",
    "model": "text-embedding-3-small"
}

4. Vectorization Script

File: scripts/vectorize.py

from qdrant_client import QdrantClient
from qdrant_client.models import PointStruct, VectorParams, Distance
from openai import OpenAI
import tiktoken
import uuid
import json
from pathlib import Path

class TextbookVectorizer:
    def __init__(self, qdrant_url, qdrant_api_key, openai_api_key):
        self.qdrant = QdrantClient(url=qdrant_url, api_key=qdrant_api_key)
        self.openai = OpenAI(api_key=openai_api_key)
        self.encoder = tiktoken.encoding_for_model("text-embedding-3-small")

    def create_collection(self, name: str, vector_size: int = 1536):
        self.qdrant.create_collection(
            collection_name=name,
            vectors_config=VectorParams(size=vector_size, distance=Distance.COSINE)
        )

    def chunk_markdown(self, content: str, source: str) -> list:
        # Split by headers for semantic coherence
        chunks = []
        current_chunk = ""
        current_section = "Introduction"

        for line in content.split("\n"):
            if line.startswith("## "):
                if current_chunk:
                    chunks.append({"content": current_chunk, "section": current_section})
                current_section = line[3:].strip()
                current_chunk = line + "\n"
            else:
                current_chunk += line + "\n"
                if len(self.encoder.encode(current_chunk)) > 500:
                    chunks.append({"content": current_chunk, "section": current_section})
                    current_chunk = ""

        if current_chunk:
            chunks.append({"content": current_chunk, "section": current_section})

        return chunks

    def embed(self, text: str) -> list:
        response = self.openai.embeddings.create(
            input=text,
            model="text-embedding-3-small"
        )
        return response.data[0].embedding

    def vectorize_file(self, filepath: Path, collection: str):
        content = filepath.read_text()
        chunks = self.chunk_markdown(content, str(filepath))

        points = []
        for i, chunk in enumerate(chunks):
            vector = self.embed(chunk["content"])
            points.append(PointStruct(
                id=str(uuid.uuid4()),
                vector=vector,
                payload={
                    "content": chunk["content"],
                    "source": str(filepath),
                    "section": chunk["section"],
                    "chunk_index": i,
                    "total_chunks": len(chunks)
                }
            ))

        self.qdrant.upsert(collection_name=collection, points=points)
        return len(points)

5. Input Variables

Usage Instructions

Step 1: Set Environment Variables

# Add to .env file
QDRANT_URL=https://your-cluster.qdrant.io
QDRANT_API_KEY=your-api-key
OPENAI_API_KEY=sk-...

Step 2: Initialize Collection

.claude/skills/qdrant-manager/scripts/setup.sh --init --collection textbook_chapters

Step 3: Vectorize Chapters

# Vectorize English chapters
.claude/skills/qdrant-manager/scripts/setup.sh --vectorize docs/

# Vectorize Urdu translations
.claude/skills/qdrant-manager/scripts/setup.sh --vectorize i18n/ur/docusaurus-plugin-content-docs/current/

Step 4: Query for RAG

# Simple query
.claude/skills/qdrant-manager/scripts/setup.sh --query "How do I install ROS 2?"

# Query with top-k
.claude/skills/qdrant-manager/scripts/setup.sh --query "Gazebo simulation" --top-k 10

Step 5: Verify Collection

# Show collection info
.claude/skills/qdrant-manager/scripts/setup.sh --info

# List all collections
.claude/skills/qdrant-manager/scripts/setup.sh --list

Verification Checklist

• QDRANT_URL environment variable set
• QDRANT_API_KEY environment variable set
• OPENAI_API_KEY environment variable set
• Collection created successfully
• Chapters vectorized without errors
• Query returns relevant results
• Metadata filtering works correctly

Integration with RAG Chatbot

This skill provides the vector storage for the RAG chatbot (Feature 002). The chatbot queries this collection to retrieve relevant context for answering user questions.

# In RAG chatbot
from qdrant_client import QdrantClient

client = QdrantClient(url=QDRANT_URL, api_key=QDRANT_API_KEY)

def get_context(query: str, language: str = "en") -> list:
    query_vector = embed(query)
    results = client.search(
        collection_name="textbook_chapters",
        query_vector=query_vector,
        query_filter={"must": [{"key": "language", "match": {"value": language}}]},
        limit=5
    )
    return [hit.payload["content"] for hit in results]

Troubleshooting

Requirements

• Python 3.9+
• Qdrant Cloud account (or self-hosted Qdrant)
• OpenAI API key (for embeddings)
• pip packages: qdrant-client, openai, tiktoken

Cost Considerations

Related

• Qdrant Documentation
• OpenAI Embeddings
• Skill: vercel-fastapi-link - Backend API with RAG endpoints
• Feature: 002-rag-chatbot - RAG chatbot specification

Changelog

v1.0.0 (2026-01-02)

Initial Release

• Collection creation and management
• Markdown chunking with header-based splitting
• OpenAI embedding integration (text-embedding-3-small)
• Metadata extraction (chapter, section, language, keywords)
• Query support with filtering
• Batch vectorization with progress tracking