# Vector Dbs

> Vector Databases for Semantic Search

- Skill: `muhammederem/vector-dbs` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add muhammederem/vector-dbs`
- Raw SKILL.md: https://api.skillmd.com/api/skills/muhammederem/vector-dbs/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: muhammederem (https://skillmd.com/u/muhammederem)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/muhammederem/vector-dbs

---

# Vector Databases for Semantic Search

## Overview
Vector databases store and query high-dimensional vector embeddings, enabling semantic search, recommendation systems, and RAG applications.

## Key Concepts

### Embeddings
Vector representations of text/images that capture semantic meaning:
- Text: 384-1536 dimensions (OpenAI, Sentence Transformers)
- Images: 512-2048 dimensions (CLIP, ResNet)
- Dense vectors vs sparse vectors

### Similarity Metrics
- **Cosine Similarity**: Angle between vectors (range: -1 to 1)
- **Euclidean Distance**: Straight-line distance
- **Dot Product**: Unnormalized cosine similarity

### Key Operations
1. **Index**: Store vectors with metadata
2. **Search**: Find nearest neighbors
3. **Delete**: Remove vectors
4. **Update**: Modify vectors or metadata

## Pinecone

### Setup
```bash
pip install pinecone-client
```

```python
import pinecone

# Initialize
pinecone.init(
    api_key="your-api-key",
    environment="us-west1-gcp"
)

# Create index
pinecone.create_index(
    name="my-index",
    dimension=1536,
    metric="cosine",
    pod_type="p1.x1"
)

# Connect
index = pinecone.Index("my-index")
```

### Basic Operations
```python
# Upsert vectors
index.upsert(
    vectors=[
        ("vec1", [0.1, 0.2, ...], {"category": "tech"}),
        ("vec2", [0.3, 0.4, ...], {"category": "news"})
    ]
)

# Query
results = index.query(
    vector=[0.1, 0.2, ...],
    top_k=10,
    include_metadata=True
)

# Delete
index.delete(ids=["vec1", "vec2"])
```

### Filtering
```python
results = index.query(
    vector=query_vector,
    filter={"category": {"$eq": "tech"}},
    top_k=10
)
```

## Weaviate

### Setup
```bash
pip install weaviate-client
```

```python
import weaviate

# Connect
client = weaviate.Client("http://localhost:8080")

# Create class
client.schema.create_class({
    "class": "Document",
    "properties": [
        {"name": "text", "dataType": ["text"]},
        {"name": "category", "dataType": ["string"]}
    ],
    "vectorizer": "text2vec-openai"
})
```

### Basic Operations
```python
# Add data object
client.data_object.create(
    class_name="Document",
    data_object={
        "text": "Sample text",
        "category": "tech"
    }
)

# Query
results = client.query.get(
    "Document",
    ["text", "category"]
).with_near_vector({
    "vector": query_vector,
    "certainty": 0.7
}).with_limit(10).do()

# Delete
client.data_object.delete(
    class_name="Document",
    uuid=obj_id
)
```

### Hybrid Search
```python
results = client.query.get(
    "Document",
    ["text"]
).with_hybrid(
    query="search terms",
    vector=query_vector,
    alpha=0.5  # 0 = keyword, 1 = vector
).with_limit(10).do()
```

## Qdrant

### Setup
```bash
pip install qdrant-client
```

```python
from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams, PointStruct

# Connect
client = QdrantClient("localhost", port=6333)

# Create collection
client.create_collection(
    collection_name="my_collection",
    vectors_config=VectorParams(size=1536, distance=Distance.COSINE)
)
```

### Basic Operations
```python
# Upsert points
client.upsert(
    collection_name="my_collection",
    points=[
        PointStruct(id=1, vector=[0.1, 0.2, ...], payload={"text": "sample"}),
        PointStruct(id=2, vector=[0.3, 0.4, ...], payload={"text": "example"})
    ]
)

# Search
results = client.search(
    collection_name="my_collection",
    query_vector=[0.1, 0.2, ...],
    limit=10,
    with_payload=True
)

# Delete
client.delete(
    collection_name="my_collection",
    points_selector=[1, 2]
)
```

### Filtering
```python
from qdrant_client.models import Filter

results = client.search(
    collection_name="my_collection",
    query_vector=query_vector,
    query_filter=Filter(
        must=[{"key": "category", "match": {"value": "tech"}}]
    )
)
```

## Embedding Generation

### OpenAI Embeddings
```python
from langchain_openai import OpenAIEmbeddings

embeddings = OpenAIEmbeddings(openai_api_key="your-key")
vector = embeddings.embed_query("Your text here")
```

### Sentence Transformers
```python
from sentence_transformers import SentenceTransformer

model = SentenceTransformer('all-MiniLM-L6-v2')
vectors = model.encode(["text1", "text2"])
```

### Hugging Face
```python
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained('sentence-transformers/all-MiniLM-L6-v2')
model = AutoModel.from_pretrained('sentence-transformers/all-MiniLM-L6-v2')

# Generate embeddings
inputs = tokenizer(texts, padding=True, truncation=True, return_tensors="pt")
embeddings = model(**inputs).last_hidden_state.mean(dim=1)
```

## Best Practices

### 1. Embedding Strategy
- **Domain-specific**: Use models fine-tuned on your domain
- **Multilingual**: Use multilingual models for international content
- **Batch processing**: Embed in batches for efficiency

### 2. Chunking Strategies
```python
# Fixed size
chunks = [text[i:i+1000] for i in range(0, len(text), 1000)]

# Semantic
from langchain.text_splitter import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200
)
```

### 3. Metadata Design
- Store relevant filtering fields
- Include timestamps, sources, categories
- Keep metadata lightweight

### 4. Index Optimization
- **Pinecone**: Choose appropriate pod type (s1 vs p1)
- **Weaviate**: Use HNSW for fast approximate search
- **Qdrant**: Tune quantization for memory efficiency

### 5. Query Optimization
```python
# Hybrid search (vector + keyword)
# Re-ranking
# Filtering before vector search
# Caching frequent queries
```

## RAG Integration

### End-to-End Pipeline
```python
from langchain_community.vectorstores import Pinecone
from langchain_openai import OpenAIEmbeddings
from langchain.chains import RetrievalQA
from langchain_openai import ChatOpenAI

# Vector store
vectorstore = Pinecone.from_documents(
    documents=documents,
    embedding=OpenAIEmbeddings(),
    index_name="rag-index"
)

# RAG chain
qa = RetrievalQA.from_chain_type(
    llm=ChatOpenAI(model="gpt-4"),
    chain_type="stuff",
    retriever=vectorstore.as_retriever(search_kwargs={"k": 3})
)
```

### Advanced RAG
```python
# Multi-query retrieval
from langchain.retrievers import MultiQueryRetriever

retriever = MultiQueryRetriever.from_llm(
    retriever=vectorstore.as_retriever(),
    llm=ChatOpenAI()
)

# Contextual compression
from langchain.retrievers import ContextualCompressionRetriever
from langchain.retrievers.document_compressors import LLMChainExtractor

compressor = LLMChainExtractor.from_llm(ChatOpenAI())
compression_retriever = ContextualCompressionRetriever(
    base_compressor=compressor,
    base_retriever=vectorstore.as_retriever()
)
```

## Performance Tuning

### Pinecone
- Use namespaces for multi-tenancy
- Batch upsert (max 100 vectors per batch)
- Choose pod type based on needs (s1 for storage, p1 for performance)

### Weaviate
- Tune HNSW parameters (ef_construction, M, ef)
- Enable replication for high availability
- Use sharding for large datasets

### Qdrant
- Enable quantization for memory savings
- Use optimizers for index building
- Tune search parameters (hnsw_ef, payload_index)

## Comparison

| Feature | Pinecone | Weaviate | Qdrant |
|---------|----------|----------|--------|
| Managed | ✓ | Self-hosted | Both |
| Open Source | ✗ | ✓ | ✓ |
| Hybrid Search | ✗ | ✓ | ✓ |
| Filtering | ✓ | ✓ | ✓ |
| Scalability | High | High | High |
| Setup | Easiest | Medium | Medium |

## Common Patterns

### Semantic Search
```python
def semantic_search(query, top_k=5):
    query_vector = embeddings.embed_query(query)
    results = index.query(vector=query_vector, top_k=top_k)
    return results
```

### Recommendation System
```python
def find_similar_items(item_id, top_k=10):
    item_vector = get_item_vector(item_id)
    results = index.query(vector=item_vector, top_k=top_k)
    return results
```

### Deduplication
```python
def find_duplicates(text, threshold=0.95):
    vector = embeddings.embed_query(text)
    results = index.query(vector=vector, top_k=10)
    duplicates = [r for r in results if r.score > threshold]
    return duplicates
```

## Integration

- **LangChain**: All vector stores supported
- **LlamaIndex**: Vector store integrations
- **Haystack**: Document stores
- **Embedding Models**: OpenAI, Cohere, Sentence Transformers