Optimization Embedding Tuning

Embedding Optimization

🎯 Trigger Conditions

Use when asked about embedding optimization, vector search tuning, or improving retrieval quality in DSPy.

📚 Prerequisites

• dspy package installed
• Embedding model available
• Training data with relevance labels

🛠️ Embedding Optimization Techniques

1. Embedding Selection

import dspy

# Different embedding models
embeddings = {
    "small": dspy.OllamaEmbedding(model="nomic-embed"),
    "medium": dspy.OpenAIEmbedding(model="text-embedding-3-small"),
    "large": dspy.OpenAIEmbedding(model="text-embedding-3-large")
}

# Choose based on needs
retriever = dspy.Retrieve(
    k=5,
    embedding_model=embeddings["medium"]
)

2. Embedding Fine-Tuning

from sentence_transformers import SentenceTransformer, InputExample, losses

# Prepare training data
train_examples = [
    InputExample(texts=["query", "relevant passage"]),
    InputExample(texts=["query", "irrelevant passage"])
]

# Create model
model = SentenceTransformer("all-MiniLM-L6-v2")

# Define loss
train_loss = losses.CosineSimilarityLoss(model)

# Fine-tune
model.fit(
    train_objectives=[(train_dataloader, train_loss)],
    epochs=3,
    warmup_steps=100
)

3. Embedding Cache Optimization

import dspy

# Configure embedding cache
dspy.settings.configure(
    rm=dspy.LMCache(
        lm=dspy.LM("openai/gpt-3.5-turbo"),
        cache_path="./embedding_cache"
    )
)

# Use cached embeddings
retriever = dspy.Retrieve(
    k=5,
    embedding_model=dspy.OllamaEmbedding(model="nomic-embed")
)

4. Hybrid Search Optimization

class HybridSearch(dspy.Module):
    def __init__(self):
        self.vector_retriever = dspy.Retrieve(k=5)
        self.keyword_retriever = dspy.BM25Retriever(k=5)
        self.reranker = dspy.Reranker(top_n=3)
    
    def forward(self, query):
        # Hybrid retrieval
        vector_results = self.vector_retriever(query).passages
        keyword_results = self.keyword_retriever(query).passages
        
        # Combine results
        combined = self.combine_results(vector_results, keyword_results)
        
        # Rerank
        return self.reranker(query, combined)
    
    def combine_results(self, vector_results, keyword_results):
        # Reciprocal rank fusion
        scores = {}
        for i, passage in enumerate(vector_results):
            scores[passage] = 1 / (i + 1)
        for i, passage in enumerate(keyword_results):
            scores[passage] = scores.get(passage, 0) + 1 / (i + 1)
        
        return sorted(scores.keys(), key=lambda x: scores[x], reverse=True)

⚠️ Pitfalls

• Dimensionality: Higher dimensions increase compute
• Bias: Embeddings can encode dataset bias
• Update frequency: Embeddings need regular updates
• Storage: Large embedding collections require storage

📖 References

• Embedding Optimization
• Sentence Transformers