Observability Elasticsearch

Elasticsearch Analysis

Authentication

IMPORTANT: Credentials are injected automatically by a proxy layer. Do NOT check for ELASTICSEARCH_URL, ES_USER, or ES_PASSWORD in environment variables - they won't be visible to you. Just run the scripts directly; authentication is handled transparently.

MANDATORY: Statistics-First Investigation

NEVER dump raw logs. Always follow this pattern:

STATISTICS → SAMPLE → PATTERNS → CORRELATE

1. Statistics First - Know volume, error rate, and top patterns before sampling
2. Strategic Sampling - Choose the right strategy based on statistics
3. Pattern Extraction - Cluster similar errors to find root causes
4. Context Correlation - Investigate around anomaly timestamps

Available Scripts

All scripts are in .claude/skills/observability-elasticsearch/scripts/

PRIMARY INVESTIGATION SCRIPTS

get_statistics.py - ALWAYS START HERE

Comprehensive statistics with pattern extraction.

python .claude/skills/observability-elasticsearch/scripts/get_statistics.py [--index INDEX] [--time-range MINUTES]

# Examples:
python .claude/skills/observability-elasticsearch/scripts/get_statistics.py --time-range 60
python .claude/skills/observability-elasticsearch/scripts/get_statistics.py --index logs-production

Output includes:

• Total count, error count, error rate percentage
• Status distribution (info, warn, error)
• Top services/sources by log volume
• Top error patterns (crucial for quick triage)
• Actionable recommendation

sample_logs.py - Strategic Sampling

Choose the right sampling strategy based on statistics.

python .claude/skills/observability-elasticsearch/scripts/sample_logs.py --strategy STRATEGY [--index INDEX] [--limit N]

# Strategies:
#   errors_only   - Only error logs (default for incidents)
#   warnings_up   - Warning and error logs
#   around_time   - Logs around a specific timestamp
#   all           - All log levels

# Examples:
python .claude/skills/observability-elasticsearch/scripts/sample_logs.py --strategy errors_only --index logs-production
python .claude/skills/observability-elasticsearch/scripts/sample_logs.py --strategy around_time --timestamp "2026-01-27T05:00:00Z" --window 5

Lucene Query Syntax

Basic Searches

# Simple term
error

# Phrase
"connection refused"

# Field search
level:ERROR

# Wildcard
message:timeout*

# Multiple terms (implicit OR)
error warning

# Required term (AND)
+error +timeout

Field Queries

# Exact match
level:ERROR

# Wildcard
host:web-*

# Range (numeric)
status:[400 TO 599]

# Range (dates)
@timestamp:[2024-01-15T10:00:00 TO 2024-01-15T11:00:00]

# Exists
_exists_:error.stack_trace

Boolean Operators

# AND
error AND timeout

# OR
error OR warning

# NOT
error NOT debug

# Grouping
(error OR warning) AND service:api

Query DSL (JSON)

Match Query

{
  "query": {
    "match": {
      "message": "connection error"
    }
  }
}

Term Query (Exact Match)

{
  "query": {
    "term": {
      "level": "ERROR"
    }
  }
}

Bool Query (Compound)

{
  "query": {
    "bool": {
      "must": [
        {"term": {"level": "ERROR"}},
        {"match": {"message": "timeout"}}
      ],
      "must_not": [
        {"term": {"service": "healthcheck"}}
      ],
      "filter": [
        {"range": {"@timestamp": {"gte": "now-1h"}}}
      ]
    }
  }
}

Aggregations

{
  "size": 0,
  "aggs": {
    "errors_by_service": {
      "terms": {
        "field": "service.keyword",
        "size": 10
      }
    }
  }
}

Investigation Workflow

Standard Incident Investigation

┌─────────────────────────────────────────────────────────────┐
│ 1. STATISTICS FIRST (mandatory)                              │
│    python get_statistics.py --index <index>                  │
│    → Know volume, error rate, top patterns                   │
└─────────────────────────────────────────────────────────────┘
                             │
                             ▼
                     High Error Rate?
               ┌─────────────┴─────────────┐
               │                           │
       YES (>5%)                           NO
               │                           │
               ▼                           ▼
┌─────────────────────────────┐  ┌───────────────────────────────────────────┐
│ 2. FAST PATH                │  │ 2. TARGETED INVESTIGATION                 │
│    Sample errors directly   │  │    Filter by specific criteria            │
│    python sample_logs.py    │  │    python sample_logs.py --strategy all   │
│    --strategy errors_only   │  │    → Look for anomalies                   │
└─────────────────────────────┘  └───────────────────────────────────────────┘

Quick Commands Reference

Common Aggregation Patterns

Errors Over Time

{
  "size": 0,
  "query": {"term": {"level": "ERROR"}},
  "aggs": {
    "errors_over_time": {
      "date_histogram": {
        "field": "@timestamp",
        "fixed_interval": "5m"
      }
    }
  }
}

Top Error Messages

{
  "size": 0,
  "query": {"term": {"level": "ERROR"}},
  "aggs": {
    "top_errors": {
      "terms": {
        "field": "message.keyword",
        "size": 10
      }
    }
  }
}

Nested Aggregation (Errors by Service, then by Message)

{
  "size": 0,
  "aggs": {
    "by_service": {
      "terms": {"field": "service.keyword", "size": 10},
      "aggs": {
        "by_message": {
          "terms": {"field": "message.keyword", "size": 5}
        }
      }
    }
  }
}

Field Types

Keyword vs Text

• keyword: Exact match, aggregatable (service.keyword)
• text: Full-text search, not aggregatable (message)

// For aggregation, use .keyword suffix
"terms": {"field": "service.keyword"}

// For full-text search, use text field
"match": {"message": "connection error"}

Anti-Patterns to Avoid

1. ❌ NEVER skip statistics - get_statistics.py is MANDATORY first step
2. ❌ Unbounded queries - Always specify time ranges and limits
3. ❌ Fetching all logs - Use sampling strategies, not unbounded searches
4. ❌ Ignoring error rate - High error rate means immediate investigation
5. ❌ Text field in aggregation - Use .keyword suffix for terms aggs
6. ❌ Wildcard prefix - *error is expensive, prefer error* or exact match

Source: swapnildahiphale/OpenSRE — distributed by TomeVault.