Logql Generator

LogQL Query Generator

Overview

Interactive workflow for generating production-ready LogQL queries. LogQL is Grafana Loki's query language—distributed grep with labels for filtering, plus aggregation and metrics capabilities.

When to Use This Skill

• Creating LogQL queries for log analysis, dashboards, or alerting
• Converting log analysis requirements into LogQL expressions
• Troubleshooting applications through log analysis
• Working with structured logs (JSON, logfmt)

Interactive Query Planning Workflow

CRITICAL: Always engage the user in collaborative planning before generating queries.

Stage 1: Understand the Goal

Gather requirements using AskUserQuestion:

1. Primary Goal: Error analysis, performance tracking, security monitoring, debugging, pattern detection?
2. Use Case: Dashboard, alerting rule, ad-hoc troubleshooting, metrics generation?
3. Context: Application/service, environment, time range, log format?

Stage 2: Identify Log Sources

1. Labels: What labels identify your logs? (job, namespace, app, level, service_name)
2. Log Format: JSON, logfmt, plain text, or custom?
3. Strategy: Use labels for stream selection (indexed), line filters for content (not indexed)

Stage 3: Determine Query Parameters

1. Query Type: Log query (return lines) or metric query (calculate values)?
2. Filtering: Stream selector {job="app"}, line filters |= "error", label filters | status >= 500
3. Parsing: | json, | logfmt, | pattern "<ip> - <user>", | regexp "(?P<field>...)"
4. Aggregation: count_over_time(), rate(), sum by (label), quantile_over_time()
5. Time Range: [5m], [1h], [24h]

Stage 4: Present Plan & Confirm

Before generating code, present a plain-English plan:

## LogQL Query Plan

**Goal**: [Description]
**Query Structure**:
1. Select streams: `{label="value"}`
2. Filter lines: [operations]
3. Parse logs: [parser]
4. Aggregate: [function]

**Does this match your intentions?**

Use AskUserQuestion to confirm before proceeding.

Stage 4a: Consult Reference Files (REQUIRED for Complex Queries)

MANDATORY: Use the Read tool to explicitly read reference files during skill execution. Do NOT rely on prior knowledge or cached information.

BEFORE generating queries, when the query involves ANY of the following, you MUST use the Read tool:

How to consult (MUST use Read tool):

# Use the Read tool with these paths during skill execution:
Read(".claude/skills/logql-generator/examples/common_queries.logql")   # For query patterns
Read(".claude/skills/logql-generator/references/best_practices.md")    # For optimization and anti-patterns

Example workflow:

1. User requests alerting rule with topk aggregation
2. MUST call: Read examples/common_queries.logql to get topk patterns
3. MUST call: Read references/best_practices.md to get alerting best practices (#19-21, #39)
4. Then generate query using patterns from the files you just read

Why this matters: Reference files contain battle-tested patterns and edge cases not covered in the skill overview. Explicit consultation during each skill execution ensures you use the latest patterns and prevents syntax errors.

Stage 5: Generate Query

Best Practices

1. Specific Stream Selectors: {namespace="prod", app="api", level="error"} not just {namespace="prod"}
2. Filter Order: Line filter → parse → label filter (fastest to slowest)
3. Parser Performance: pattern > logfmt > json > regexp

Core Query Patterns

Log Filtering:

{job="app"} |= "error" |= "timeout"        # Contains both
{job="app"} |~ "error|fatal|critical"       # Regex match
{job="app"} != "debug"                      # Exclude

JSON/logfmt Parsing:

{app="api"} | json | level="error" | status_code >= 500
{app="app"} | logfmt | caller="database.go"

Pattern Extraction:

{job="nginx"} | pattern "<ip> - - [<_>] \"<method> <path>\" <status> <size>"

Metrics:

# Rate
rate({job="app"} | json | level="error" [5m])

# Count by label
sum by (app) (count_over_time({namespace="prod"} | json [5m]))

# Error percentage
sum(rate({app="api"} | json | level="error" [5m])) / sum(rate({app="api"}[5m])) * 100

# Latency percentiles
quantile_over_time(0.95, {app="api"} | json | unwrap duration [5m])

# Top N
topk(10, sum by (error_type) (count_over_time({job="app"} | json | level="error" [1h])))

Formatting:

{job="app"} | json | line_format "{{.level}}: {{.message}}"
{job="app"} | json | label_format env="{{.environment}}"

IP Filtering (prefer label filter after parsing for precision):

{job="nginx"} | logfmt | remote_addr = ip("192.168.4.0/24")

Stage 5a: Incremental Query Building (Educational/Debugging)

When to use this stage:

• User is learning LogQL
• Complex multi-stage queries
• Debugging query issues
• User explicitly requests step-by-step explanation

Present the query construction incrementally:

## Building Your Query Step-by-Step

### Step 1: Stream Selector (verify logs exist)
```logql
{app="api"}

Test this first to confirm logs are flowing

Step 2: Add Line Filter (fast pre-filtering)

{app="api"} |= "error"

Reduces data before parsing

Step 3: Add Parser (extract fields)

{app="api"} |= "error" | json

Now you can filter on extracted labels

Step 4: Add Label Filter (precise filtering)

{app="api"} |= "error" | json | level="error"

Final filter on parsed data

Step 5: Add Aggregation (if metric query)

sum(count_over_time({app="api"} |= "error" | json | level="error" [5m]))

Complete metric query

**Benefits of incremental building:**
1. Identify which step breaks (no results, parse errors)
2. Understand performance impact of each operation
3. Debug unexpected results by testing each stage
4. Learn LogQL query structure naturally

**Use AskUserQuestion** to offer incremental mode:
- Option: "Show step-by-step construction" vs "Show final query only"

### Stage 6: Provide Usage

1. **Final Query** with explanation
2. **How to Use**: Grafana panel, Loki alerting rules, `logcli query`, HTTP API
3. **Customization**: Labels to modify, thresholds to tune

## Advanced Techniques

### Multiple Parsers
```logql
{app="api"} | json | regexp "user_(?P<user_id>\\d+)"

Unwrap for Numeric Metrics

sum(sum_over_time({app="api"} | json | unwrap duration [5m]))

Pattern Match Operators (Loki 3.0+, 10x faster than regex)

{service_name=`app`} |> "<_> level=debug <_>"

Logical Operators

{app="api"} | json | (status_code >= 400 and status_code < 500) or level="error"

Offset Modifier

sum(rate({app="api"} | json | level="error" [5m])) - sum(rate({app="api"} | json | level="error" [5m] offset 1d))

Label Operations

{app="api"} | json | keep namespace, pod, level
{app="api"} | json | drop pod, instance

Note: LogQL has no dedup or distinct operators. Use metric aggregations like sum by (field) for programmatic deduplication.

Loki 3.x Key Features

Structured Metadata

High-cardinality data without indexing (trace_id, user_id, request_id):

# Filter AFTER stream selector, NOT in it
{app="api"} | trace_id="abc123" | json | level="error"

Query Acceleration (Bloom Filters)

Place structured metadata filters BEFORE parsers:

# ACCELERATED
{cluster="prod"} | detected_level="error" | logfmt | json
# NOT ACCELERATED
{cluster="prod"} | logfmt | json | detected_level="error"

approx_topk (Probabilistic)

approx_topk(10, sum by (endpoint) (rate({app="api"}[5m])))

vector() for Alerting

sum(count_over_time({app="api"} | json | level="error" [5m])) or vector(0)

Automatic Labels

• service_name: Auto-populated from container name
• detected_level: Auto-detected when discover_log_levels: true (stored as structured metadata)

Function Reference

Log Range Aggregations

Unwrapped Range Aggregations

Aggregation Operators

sum, avg, min, max, count, stddev, topk, bottomk, approx_topk, sort, sort_desc

With grouping: sum by (label1, label2) or sum without (label1)

Conversion Functions

label_replace()

label_replace(rate({job="api"} |= "err" [1m]), "foo", "$1", "service", "(.*):.*")

Parser Reference

logfmt

| logfmt [--strict] [--keep-empty]

• --strict: Error on malformed entries
• --keep-empty: Keep standalone keys

JSON

| json                                           # All fields
| json method="request.method", status="response.status"  # Specific fields
| json servers[0], headers="request.headers[\"User-Agent\"]"  # Nested/array

Template Functions

Common functions for line_format and label_format:

String: trim, upper, lower, replace, trunc, substr, printf, contains, hasPrefix Math: add, sub, mul, div, addf, subf, floor, ceil, round Date: date, now, unixEpoch, toDate, duration_seconds Regex: regexReplaceAll, count Other: fromJson, default, int, float64, __line__, __timestamp__

See examples/common_queries.logql for detailed usage.

Alerting Rules

# Alert when error rate exceeds 5%
(sum(rate({app="api"} | json | level="error" [5m])) / sum(rate({app="api"}[5m]))) > 0.05

# With vector() to avoid "no data"
sum(rate({app="api"} | json | level="error" [5m])) or vector(0) > 10

Error Handling

Documentation Lookup

When to Fetch External Documentation (MANDATORY)

Trigger context7 MCP or WebSearch when the query involves ANY of these:

How to Use

context7 MCP (preferred - authoritative docs):

1. mcp__context7__resolve-library-id with libraryName="grafana loki"
2. mcp__context7__get-library-docs with context7CompatibleLibraryID and topic="[specific topic]"

WebSearch (fallback for latest features):

WebSearch query: "Grafana Loki LogQL [topic] documentation [year]"

Example Workflow

When user asks for "error tracking with trace correlation in Loki 3.x":

1. Recognize trigger: "Loki 3.x" + "trace" → structured metadata
2. Fetch docs: mcp__context7__get-library-docs with topic="structured metadata trace_id"
3. Apply patterns from docs to generate accurate query

Resources

• examples/common_queries.logql: Comprehensive query examples
• references/best_practices.md: 39+ LogQL best practices, performance optimization, anti-patterns

Guidelines

1. Always plan interactively - Present plain-English plan before generating
2. Use AskUserQuestion - Gather requirements and confirm plans
3. MUST use Read tool for complex queries - Explicitly call Read on examples/common_queries.logql and references/best_practices.md during skill execution for alerting rules, topk, percentiles, or performance-critical queries. Do NOT skip this step or rely on prior knowledge.
4. Fetch docs for advanced features - Use context7 MCP when Loki 3.x features, approx_topk, or unclear syntax is involved (see Documentation Lookup triggers)
5. Offer incremental building - For learning or debugging, present step-by-step query construction (see Stage 5a)
6. Explain queries - What it does, how to interpret results
7. Prioritize performance - Specific selectors, filter early, simpler parsers

Version Notes

• Loki 3.0+: Bloom filters, structured metadata, pattern match operators (|>, !>)
• Loki 3.3+: approx_topk function
• Loki 3.5+: Promtail deprecated (use Grafana Alloy)
• Loki 3.6+: Horizontally scalable compactor, Loki UI as Grafana plugin

Deprecations: Promtail (use Alloy), BoltDB store (use TSDB with v13 schema)