Search Engineer

Search Engineer

Purpose

Design and implement search systems that deliver fast, relevant results. This skill covers search engine selection, indexing strategies, relevance tuning, autocomplete, faceted search, and operational monitoring.

Search Engine Selection

Decision Matrix

ENGINE          BEST FOR                  COMPLEXITY   COST
------------------------------------------------------------
PostgreSQL FTS  Simple search, < 1M docs  Low          Free
Meilisearch     Typo-tolerant, instant    Low          Free/Hosted
Typesense       Fast, easy setup          Low          Free/Hosted
Elasticsearch   Complex queries, logs     High         Resource-heavy
OpenSearch      AWS ecosystem             High         Resource-heavy
Algolia         Hosted, instant UX        Low          Pay per search

DECISION TREE:
  Do you need search beyond simple LIKE queries?
    NO  -> PostgreSQL LIKE with trigram index
    YES -> How many documents?
      < 100K -> Meilisearch or Typesense
      100K - 10M -> Any engine (based on feature needs)
      > 10M -> Elasticsearch / OpenSearch

  Do you need:
    Log aggregation + search? -> Elasticsearch
    Instant autocomplete?     -> Meilisearch or Typesense
    Complex aggregations?     -> Elasticsearch
    Simple full-text?         -> PostgreSQL FTS
    Hosted with zero ops?     -> Algolia

PostgreSQL Full-Text Search

Basic Setup

-- Add tsvector column for search
ALTER TABLE products ADD COLUMN search_vector tsvector;

-- Populate search vector with weighted fields
UPDATE products SET search_vector =
  setweight(to_tsvector('english', coalesce(name, '')), 'A') ||
  setweight(to_tsvector('english', coalesce(description, '')), 'B') ||
  setweight(to_tsvector('english', coalesce(category, '')), 'C');

-- Create GIN index for fast search
CREATE INDEX idx_products_search ON products USING gin(search_vector);

-- Auto-update trigger
CREATE OR REPLACE FUNCTION products_search_trigger() RETURNS trigger AS $$
BEGIN
  NEW.search_vector :=
    setweight(to_tsvector('english', coalesce(NEW.name, '')), 'A') ||
    setweight(to_tsvector('english', coalesce(NEW.description, '')), 'B') ||
    # ... (condensed) ...
  ts_headline('english', description, query,
    'StartSel=<mark>, StopSel=</mark>, MaxWords=35, MinWords=15'
  ) AS highlighted_description
FROM products, plainto_tsquery('english', 'wireless bluetooth headphones') AS query
WHERE search_vector @@ query
ORDER BY rank DESC
LIMIT 20;

Trigram Search (Fuzzy Matching)

-- Enable extension
CREATE EXTENSION IF NOT EXISTS pg_trgm;

-- Create trigram index
CREATE INDEX idx_products_name_trgm ON products USING gin(name gin_trgm_ops);

-- Fuzzy search with similarity score
SELECT name, similarity(name, 'blutooth hedphones') AS sim
FROM products
WHERE name % 'blutooth hedphones'
ORDER BY sim DESC
LIMIT 10;

-- Combine FTS with trigram for best results
SELECT id, name,
  ts_rank(search_vector, tsquery) * 2 + similarity(name, :query) AS combined_score
FROM products, plainto_tsquery('english', :query) AS tsquery
WHERE search_vector @@ tsquery OR name % :query
ORDER BY combined_score DESC
LIMIT 20;

Elasticsearch Patterns

Index Mapping

{
  "mappings": {
    "properties": {
      "name": {
        "type": "text",
        "analyzer": "english",
        "fields": {
          "keyword": { "type": "keyword" },
          "autocomplete": {
            "type": "text",
            "analyzer": "autocomplete_analyzer",
            "search_analyzer": "standard"
          }
        }
      },
      "description": {
        "type": "text",
        "analyzer": "english"
      # ... (condensed) ...
        }
      }
    },
    "number_of_shards": 2,
    "number_of_replicas": 1
  }
}

Search Query DSL

// Multi-match search with boosting
const searchQuery = {
  query: {
    bool: {
      must: [
        {
          multi_match: {
            query: userQuery,
            fields: ['name^3', 'description^1', 'tags^2'],
            type: 'best_fields',
            fuzziness: 'AUTO',
            prefix_length: 2,
          },
        },
      ],
      filter: [
        ...(category ? [{ term: { category } }] : []),
        ...(priceMin || priceMax ? [{
          # ... (condensed) ...
  from: (page - 1) * pageSize,
  size: pageSize,
  sort: [
    { _score: 'desc' },
    { created_at: 'desc' },
  ],
};

Faceted Search (Aggregations)

const searchWithFacets = {
  query: { /* ... search query ... */ },
  aggs: {
    categories: {
      terms: { field: 'category', size: 20 },
    },
    price_ranges: {
      range: {
        field: 'price',
        ranges: [
          { to: 25, key: 'under_25' },
          { from: 25, to: 50, key: '25_to_50' },
          { from: 50, to: 100, key: '50_to_100' },
          { from: 100, key: 'over_100' },
        ],
      },
    },
    avg_price: { avg: { field: 'price' } },
    # ... (condensed) ...
  aggregations: {
    categories: { buckets: Array<{ key: string; doc_count: number }> };
    price_ranges: { buckets: Array<{ key: string; doc_count: number }> };
    avg_price: { value: number };
    tags: { buckets: Array<{ key: string; doc_count: number }> };
  };
}

Meilisearch / Typesense Patterns

Meilisearch Setup

import { MeiliSearch } from 'meilisearch';

const client = new MeiliSearch({
  host: '[reference URL]',
  apiKey: 'masterKey',
});

// Create index with settings
const index = client.index('products');

await index.updateSettings({
  searchableAttributes: ['name', 'description', 'category', 'tags'],
  filterableAttributes: ['category', 'price', 'tags', 'inStock'],
  sortableAttributes: ['price', 'createdAt', 'popularity'],
  rankingRules: [
    'words',
    'typo',
    'proximity',
    # ... (condensed) ...
  sort: ['price:asc'],
  limit: 20,
  offset: 0,
  attributesToHighlight: ['name', 'description'],
  highlightPreTag: '<mark>',
  highlightPostTag: '</mark>',
});

Autocomplete Implementation

Search-as-You-Type

// Backend: dedicated autocomplete endpoint
app.get('/api/search/suggest', async (req, res) => {
  const query = req.query.q as string;
  if (!query || query.length < 2) return res.json({ suggestions: [] });

  const suggestions = await index.search(query, {
    limit: 7,
    attributesToRetrieve: ['id', 'name', 'category'],
    attributesToHighlight: ['name'],
  });

  res.json({
    suggestions: suggestions.hits.map(hit => ({
      id: hit.id,
      text: hit.name,
      highlighted: hit._formatted?.name,
      category: hit.category,
    })),
  # ... (condensed) ...
      .catch(() => {});

    return () => controller.abort();
  }, [debouncedQuery]);

  return { query, setQuery, suggestions };
}

Query Suggestions (Popular and Recent)

// Track search queries for popularity
async function trackSearch(query: string, userId?: string) {
  const normalized = query.toLowerCase().trim();
  await redis.zincrby('search:popular', 1, normalized);
  if (userId) {
    await redis.lpush(`search:recent:${userId}`, normalized);
    await redis.ltrim(`search:recent:${userId}`, 0, 19);
  }
}

// Get query suggestions
async function getQuerySuggestions(prefix: string, userId?: string) {
  const suggestions: QuerySuggestion[] = [];

  // Recent searches (user-specific)
  if (userId) {
    const recent = await redis.lrange(`search:recent:${userId}`, 0, 4);
    const matching = recent.filter(q => q.startsWith(prefix.toLowerCase()));
    # ... (condensed) ...
  const matching = popular
    .filter(q => q.startsWith(prefix.toLowerCase()))
    .slice(0, 5);
  suggestions.push(...matching.map(q => ({ type: 'popular' as const, text: q })));

  return suggestions;
}

Relevance Scoring

Boosting Strategy

FIELD BOOSTING:
  Title/Name:      3x boost (most relevant)
  Tags/Keywords:   2x boost
  Description:     1x boost (default)
  Comments/Notes:  0.5x boost

FRESHNESS BOOST:
  Items < 7 days:   1.5x boost
  Items < 30 days:  1.2x boost
  Older items:      1x (no boost)

POPULARITY BOOST:
  Based on view count, sales, ratings
  Logarithmic scale to prevent runaway popular items

PERSONALIZATION:
  Boost items in user's preferred categories
  Boost items from followed brands/authors
  Demote previously viewed items (freshness)

Search Quality Metrics

KEY METRICS TO MONITOR:

Click-Through Rate (CTR):
  Percentage of searches resulting in at least one click.
  Target: > 60%

Mean Reciprocal Rank (MRR):
  1 / position_of_first_clicked_result, averaged.
  Target: > 0.5 (users typically click top 2 results)

Zero Result Rate:
  Percentage of searches returning no results.
  Target: < 5%

Refinement Rate:
  Percentage of searches followed by another search (indicates poor results).
  Target: < 30%

Time to First Click:
  How quickly users find and click a relevant result.
  Target: < 5 seconds

Indexing Strategy

Keeping Index in Sync

STRATEGY 1: Real-time sync (event-driven)
  Database change -> Event -> Update search index
  Latency: < 1 second
  Best for: User-facing search requiring freshness

STRATEGY 2: Batch sync (scheduled)
  Cron job -> Query changed records -> Update search index
  Latency: Minutes to hours
  Best for: Catalog search, analytics

STRATEGY 3: Change Data Capture (CDC)
  Database WAL -> Debezium -> Search index
  Latency: < 5 seconds
  Best for: High-volume, reliable sync without application changes

// Event-driven sync example
class ProductSearchSync {
  @OnEvent('product.created')
  async onCreated(product: Product) {
    await this.searchClient.index('products').addDocuments([
      this.mapToDocument(product),
    ]);
  }

  @OnEvent('product.updated')
  async onUpdated(product: Product) {
    await this.searchClient.index('products').updateDocuments([
      this.mapToDocument(product),
    ]);
  }

  @OnEvent('product.deleted')
  async onDeleted(productId: string) {
    # ... (condensed) ...
      tags: product.tags.map(t => t.name),
      inStock: product.stockCount > 0,
      popularity: product.viewCount + product.salesCount * 10,
      createdAt: product.createdAt.getTime(),
    };
  }
}

Index Rebuilding

// Full reindex strategy (zero-downtime)
async function reindexProducts() {
  const newIndex = `products_${Date.now()}`;

  // 1. Create new index with updated settings
  await searchClient.createIndex(newIndex, { primaryKey: 'id' });
  await searchClient.index(newIndex).updateSettings(indexSettings);

  // 2. Index all documents in batches
  let offset = 0;
  const batchSize = 1000;

  while (true) {
    const products = await db.products.findMany({
      skip: offset,
      take: batchSize,
      include: { category: true, tags: true },
    });
# ... (condensed) ...
  await searchClient.swapIndexes([
    { indexes: ['products', newIndex] },
  ]);

  // 4. Delete old index
  // (old index is now at the new name after swap)
}

Search Architecture Checklist

• Search engine selected based on scale and feature requirements
• Index mapping configured with proper field types and analyzers
• Searchable, filterable, and sortable attributes defined
• Autocomplete implemented with edge ngrams or search engine feature
• Faceted search configured for filter UI
• Relevance tuning applied (field boosting, freshness, popularity)
• Fuzzy matching enabled for typo tolerance
• Query suggestions from popular and recent searches
• Index sync strategy defined (real-time, batch, or CDC)
• Highlighting configured for search result previews
• Pagination implemented (offset or search-after)
• Search analytics tracking (CTR, zero results, refinements)
• Rate limiting on search endpoints
• Zero-downtime reindex strategy documented
• Monitoring covers query latency, index size, sync lag

When to Use

Use this skill when:

• Designing or implementing search engineer solutions
• Reviewing or improving existing search engineer approaches
• Making architectural or implementation decisions about search engineer
• Learning search engineer patterns and best practices
• Troubleshooting search engineer-related issues

Do NOT use this skill when:

• The question is about a fundamentally different technology domain
• A more specific sibling skill covers the exact topic needed
• The user needs a complete hands-on tutorial rather than expert guidance

Output Format

# Search Engineer Analysis

## Context Assessment
[Situation summary and constraints]

## Recommended Approach
[Primary recommendation with rationale]

## Implementation Steps
1. [Step with specific details]
2. [Step with specific details]
3. [Step with specific details]

## Trade-offs and Considerations
- [Key trade-off 1]
- [Key trade-off 2]

## Next Steps
- [Immediate action item]
- [Follow-up action item]

Example

Input: "Help me implement search engineer for a medium-scale production application"

Output: A structured analysis covering current state assessment, recommended search engineer approach with specific patterns, implementation roadmap with milestones, and risk mitigation strategies tailored to the application scale and constraints.

Edge Cases

• Legacy system integration: When search engineer must coexist with legacy approaches, provide a gradual migration path rather than a complete rewrite
• Scale mismatch: When the solution complexity exceeds the project scale, recommend a simpler approach and note when to revisit
• Team skill gaps: When the team lacks experience with the recommended approach, include learning resources and simpler alternatives
• Conflicting requirements: When constraints conflict (e.g., performance vs. maintainability), explicitly state the trade-off and recommend based on stated priorities