Elasticsearch Engineer

Elasticsearch Engineer

Core Philosophy

Elasticsearch is a distributed search and analytics engine built on Apache Lucene. It excels at full-text search, log analytics, and real-time aggregations. Success requires understanding how data flows from ingestion through analysis to inverted indexes, and how to design mappings and queries that leverage this architecture.

Index Design

Mapping Definition

PUT /products
{
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 1,
    "index.codec": "best_compression",
    "analysis": {
      "analyzer": {
        "product_analyzer": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": ["lowercase", "asciifolding", "product_synonyms", "english_stemmer"]
        },
        "autocomplete_analyzer": {
          # ... (condensed) ...
        "type": "boolean"
      }
    }
  }
}

Mapping Type Selection

When to Use Nested vs Object

// PROBLEM with object type: cross-object matching
// If you store: [{"color": "red", "size": "L"}, {"color": "blue", "size": "S"}]
// A query for color=red AND size=S would INCORRECTLY match (cross-object)

// SOLUTION: Use nested type
"attributes": {
  "type": "nested",
  "properties": {
    "color": { "type": "keyword" },
    "size": { "type": "keyword" }
  }
}

// Query with nested query
# ... (condensed) ...
        }
      }
    }
  }
}

Analyzer Configuration

Anatomy of an Analyzer

An analyzer consists of: Character Filters -> Tokenizer -> Token Filters

// Custom analyzer for product search
{
  "analysis": {
    "char_filter": {
      "html_strip": { "type": "html_strip" },
      "ampersand_replace": {
        "type": "pattern_replace",
        "pattern": "&",
        "replacement": "and"
      }
    },
    "tokenizer": {
      "product_tokenizer": {
        "type": "pattern",
        # ... (condensed) ...
        "filter": ["lowercase", "english_possessive", "english_stop", "english_stemmer"]
      }
    }
  }
}

Testing Analyzers

// Test how text is analyzed
POST /_analyze
{
  "analyzer": "english_product",
  "text": "The quick brown fox's running & jumping"
}
// Result tokens: ["quick", "brown", "fox", "run", "and", "jump"]

// Test with specific index analyzer
POST /products/_analyze
{
  "field": "name",
  "text": "Wireless Bluetooth Mouse"
}

Search Queries

Bool Query (The Foundation)

GET /products/_search
{
  "query": {
    "bool": {
      "must": [
        { "multi_match": {
          "query": "wireless mouse",
          "fields": ["name^3", "description", "tags^2"],
          "type": "best_fields",
          "fuzziness": "AUTO"
        }}
      ],
      "filter": [
        { "term": { "category": "electronics" } },
        # ... (condensed) ...
  },
  "from": 0,
  "size": 20,
  "_source": ["name", "price", "category", "in_stock"]
}

Multi-Match Strategies

// best_fields: Score from best matching field (default)
{ "multi_match": { "query": "q", "fields": ["title^3", "body"], "type": "best_fields" } }

// most_fields: Sum scores from all matching fields
{ "multi_match": { "query": "q", "fields": ["title", "title.stemmed", "title.ngram"], "type": "most_fields" } }

// cross_fields: Treats all fields as one big field (good for name = first + last)
{ "multi_match": { "query": "John Smith", "fields": ["first_name", "last_name"], "type": "cross_fields" } }

// phrase_prefix: Autocomplete / search-as-you-type
{ "multi_match": { "query": "wire mou", "fields": ["name"], "type": "phrase_prefix" } }

Function Score (Custom Relevance)

GET /products/_search
{
  "query": {
    "function_score": {
      "query": { "multi_match": { "query": "laptop", "fields": ["name", "description"] } },
      "functions": [
        {
          "field_value_factor": {
            "field": "popularity",
            "factor": 1.2,
            "modifier": "log1p",
            "missing": 1
          },
          "weight": 2
        # ... (condensed) ...
      "boost_mode": "multiply",
      "max_boost": 10
    }
  }
}

Vector Search (kNN)

// Define dense vector field
PUT /embeddings
{
  "mappings": {
    "properties": {
      "content_vector": {
        "type": "dense_vector",
        "dims": 768,
        "index": true,
        "similarity": "cosine"
      }
    }
  }
}
# ... (condensed) ...
    "k": 10,
    "num_candidates": 100
  },
  "fields": ["title", "content"]
}

Aggregations

Metric Aggregations

GET /orders/_search
{
  "size": 0,
  "aggs": {
    "total_revenue": { "sum": { "field": "total" } },
    "avg_order_value": { "avg": { "field": "total" } },
    "order_stats": {
      "stats": { "field": "total" }
    },
    "percentile_values": {
      "percentiles": { "field": "response_time_ms", "percents": [50, 90, 95, 99] }
    },
    "unique_customers": {
      "cardinality": { "field": "customer_id", "precision_threshold": 3000 }
    }
  }
}

Bucket Aggregations

GET /orders/_search
{
  "size": 0,
  "aggs": {
    "by_status": {
      "terms": { "field": "status", "size": 20 },
      "aggs": {
        "revenue": { "sum": { "field": "total" } },
        "avg_total": { "avg": { "field": "total" } }
      }
    },
    "orders_over_time": {
      "date_histogram": {
        "field": "created_at",
        # ... (condensed) ...
        ]
      }
    }
  }
}

Index Lifecycle Management (ILM)

PUT _ilm/policy/logs_policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_primary_shard_size": "50gb",
            "max_age": "1d"
          },
          "set_priority": { "priority": 100 }
        }
      },
      # ... (condensed) ...
      "number_of_shards": 3,
      "number_of_replicas": 1
    }
  }
}

Cluster Sizing

Shard Sizing Guidelines

• Target shard size: 10-50 GB per shard
• Max shards per node: ~600-1000 (depends on heap)
• Heap per shard: ~1-2 MB overhead per shard
• JVM heap: 50% of RAM, max 31 GB (compressed oops threshold)
• Rule of thumb: number_of_shards = ceil(expected_data_size / 30GB)

Node Roles

# Master-eligible nodes (3 minimum, small instances)
node.roles: [master]
# Heap: 1-4 GB, no storage needed

# Hot data nodes (high CPU, fast SSD storage)
node.roles: [data_hot, data_content]
# Heap: 16-31 GB, fast NVMe storage

# Warm data nodes (moderate CPU, larger cheaper storage)
node.roles: [data_warm]
# Heap: 16-31 GB, standard SSD storage

# Cold data nodes (minimal CPU, large HDD storage)
node.roles: [data_cold]
# Heap: 8-16 GB, HDD storage

# Coordinating-only nodes (query routing / aggregation)
node.roles: []
# Heap: 16-31 GB, no storage needed

# ML nodes
node.roles: [ml, remote_cluster_client]

Performance Tuning

Indexing Performance

// Bulk indexing settings (revert after bulk load)
PUT /my_index/_settings
{
  "index": {
    "refresh_interval": "30s",
    "number_of_replicas": 0,
    "translog.durability": "async",
    "translog.sync_interval": "30s"
  }
}

// Use bulk API (optimal batch: 5-15 MB per request)
POST /_bulk
{"index":{"_index":"products","_id":"1"}}
{"name":"Widget","price":9.99}
{"index":{"_index":"products","_id":"2"}}
{"name":"Gadget","price":19.99}

Search Performance

// Use filter context for non-scoring clauses (cacheable)
// Use routing to target specific shards
GET /orders/_search?routing=customer_123
{
  "query": {
    "bool": {
      "filter": [
        { "term": { "customer_id": "customer_123" } },
        { "range": { "created_at": { "gte": "2025-01-01" } } }
      ]
    }
  }
}

// Profile slow queries
GET /products/_search
{
  "profile": true,
  "query": { "match": { "name": "wireless mouse" } }
}

Monitoring Queries

// Cluster health
GET _cluster/health?level=indices

// Hot threads
GET _nodes/hot_threads

// Pending tasks
GET _cluster/pending_tasks

// Shard allocation explanation
GET _cluster/allocation/explain

// Index stats
GET /my_index/_stats

// Node stats
GET _nodes/stats/jvm,os,process

Kibana Dashboard Design

Effective Dashboard Layout

1. Top row: Key KPIs (metric visualizations) -- total count, error rate, p95 latency
2. Second row: Time-series line charts showing trends
3. Third row: Breakdown charts -- bar charts by category, pie charts for distribution
4. Bottom row: Data table with top items, and a saved search for drill-down

Index Patterns and Runtime Fields

// Runtime field for computed values without re-indexing
PUT /orders/_mapping
{
  "runtime": {
    "profit_margin": {
      "type": "double",
      "script": "(doc['revenue'].value - doc['cost'].value) / doc['revenue'].value * 100"
    },
    "day_of_week": {
      "type": "keyword",
      "script": "emit(doc['created_at'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))"
    }
  }
}

When to Use

Use this skill when:

• Designing or implementing elasticsearch engineer solutions
• Reviewing or improving existing elasticsearch engineer approaches
• Making architectural or implementation decisions about elasticsearch engineer
• Learning elasticsearch engineer patterns and best practices
• Troubleshooting elasticsearch engineer-related issues

Do NOT use this skill when:

• The question is about a fundamentally different technology domain
• A more specific sibling skill covers the exact topic needed
• The user needs a complete hands-on tutorial rather than expert guidance

Output Format

# Elasticsearch Engineer Analysis

## Context Assessment
[Situation summary and constraints]

## Recommended Approach
[Primary recommendation with rationale]

## Implementation Steps
1. [Step with specific details]
2. [Step with specific details]
3. [Step with specific details]

## Trade-offs and Considerations
- [Key trade-off 1]
- [Key trade-off 2]

## Next Steps
- [Immediate action item]
- [Follow-up action item]

Example

Input: "Help me implement elasticsearch engineer for a medium-scale production application"

Output: A structured analysis covering current state assessment, recommended elasticsearch engineer approach with specific patterns, implementation roadmap with milestones, and risk mitigation strategies tailored to the application scale and constraints.

Edge Cases

• Legacy system integration: When elasticsearch engineer must coexist with legacy approaches, provide a gradual migration path rather than a complete rewrite
• Scale mismatch: When the solution complexity exceeds the project scale, recommend a simpler approach and note when to revisit
• Team skill gaps: When the team lacks experience with the recommended approach, include learning resources and simpler alternatives
• Conflicting requirements: When constraints conflict (e.g., performance vs. maintainability), explicitly state the trade-off and recommend based on stated priorities