[H1][LOGQL-GENERATOR]
Dictum: Pipeline order determines query performance.
Generate LogQL for Loki 3.0-3.6 (bloom filters, structured metadata, approx_topk). Cross-references: observability-stack for Loki deployment, promql-generator for PromQL metric queries.
Tasks:
- Goal -- Error analysis, performance, security, debugging? Dashboard, alert, ad-hoc?
- Sources -- Labels (
job, namespace, app), log format (JSON/logfmt/plain), time range.
- Query type -- Log query (return lines) or metric query (calculate values)?
- Plan -- Plain-English plan, confirm with user before generating.
- Generate -- Apply patterns below; consult
references/best_practices.md.
- Deliver -- Final query + explanation + usage context (Grafana panel, alert rule, logcli, HTTP API).
[1][VERSION_MATRIX]
Dictum: Version awareness prevents deprecated patterns.
| [INDEX] |
[VERSION] |
[KEY_CHANGES] |
| [1] |
3.0 |
Bloom filters, structured metadata, pattern match |> / !>. |
| [2] |
3.3 |
Bloom acceleration for structured metadata filters. |
| [3] |
3.5 |
Promtail deprecated (EOL March 2, 2026 -- use Alloy), SSD mode deprecated. |
| [4] |
3.6 |
approx_topk on querier, reduced JSON/logfmt parser allocations, Loki UI plugin. |
Guidance:
- Promtail EOL March 2, 2026 -- migrate to Alloy.
- BoltDB deprecated -- use TSDB with v13 schema.
[2][PIPELINE_ORDER]
Dictum: Each stage filters BEFORE the next -- moving expensive operations earlier wastes resources.
{stream} -> line filter -> decolorize -> struct metadata -> parser -> label filter -> keep/drop -> format -> aggregate
cheapest most expensive
Guidance:
- Structured metadata filters BEFORE parsers enables bloom filter acceleration (3.3+).
- Line filters (
|=, !=) are O(1) substring checks -- place before parsers.
- Pattern match
|> / !> uses <_> wildcards, 10x faster than regex (3.0+).
Best-Practices:
- Extract only needed JSON fields:
| json level, status not | json -- reduces allocations (3.6).
| decolorize before | logfmt -- ANSI codes break key=value parsing.| drop instance, pod before aggregation -- reduces series cardinality.
[3][STREAM_AND_FILTERS]
Dictum: Specific stream selectors minimize chunk scanning.
| [INDEX] |
[OPERATOR] |
[MEANING] |
[EXAMPLE] |
| [1] |
|= |
Contains. |
{job="app"} |= "error" |
| [2] |
!= |
Not contains. |
{job="app"} != "debug" |
| [3] |
|~ |
Regex match. |
{job="app"} |~ "error|fatal" |
| [4] |
!~ |
Regex not match. |
{job="app"} !~ "health|metrics" |
| [5] |
|> |
Pattern match. |
{app="api"} |> "<_> level=error <_>" |
| [6] |
!> |
Pattern not match. |
{app="api"} !> "<_> level=debug <_>" |
[4][PARSERS]
Dictum: Parser selection affects per-line cost.
| [INDEX] |
[PARSER] |
[SYNTAX] |
[USE_WHEN] |
| [1] |
pattern |
| pattern "<ip> - <_> <status>" |
Fixed-delimiter structured text. |
| [2] |
logfmt |
| logfmt [--strict] [--keep-empty] |
key=value pairs. |
| [3] |
json |
| json or | json status="response.code" |
JSON (specify fields for perf). |
| [4] |
regexp |
| regexp "(?P<field>\\w+)" |
Complex extraction, last resort. |
| [5] |
unpack |
| unpack |
Packed JSON from Alloy/Promtail. |
[5][AGGREGATIONS]
Dictum: Metric queries aggregate log entries into time series.
[5.1][LOG_RANGE]
| [INDEX] |
[FUNCTION] |
[USE_WHEN] |
| [1] |
rate(log-range) |
Dashboard rate panels. |
| [2] |
count_over_time(log-range) |
Counting occurrences. |
| [3] |
bytes_rate(log-range) |
Bandwidth monitoring. |
| [4] |
absent_over_time(log-range) |
Dead service alerting. |
[5.2][UNWRAPPED_RANGE]
| [INDEX] |
[FUNCTION] |
[USE_WHEN] |
| [1] |
sum/avg/max/min_over_time |
Aggregate numeric values. |
| [2] |
quantile_over_time(phi, range) |
Latency percentiles from logs. |
| [3] |
first/last_over_time |
Boundary values. |
| [4] |
rate_counter(range) |
Counter-like log values. |
Operators: sum, avg, min, max, count, stddev, topk, bottomk, approx_topk, sort, sort_desc.
Grouping: sum by (label1, label2) (...) or sum without (label1) (...).
[5.3][APPROX_TOPK]
approx_topk(k, expr) -- probabilistic top-K via count-min sketch, instant queries only. Requires limits_config.shard_aggregations: [approx_topk] on querier.
[6][STRUCTURED_METADATA]
Dictum: Structured metadata enables high-cardinality filtering without index impact.
Filter AFTER stream selector, BEFORE parsers for bloom acceleration. NOT indexed -- no cardinality impact.
{app="api"} | trace_id="abc123" | json | level="error" # CORRECT -- bloom accelerated
# WRONG: {app="api", trace_id="abc123"} # trace_id is NOT an indexed label
Guidance:
- Bloom filters (3.3+) provide O(1) chunk skipping for string equality and OR filters placed before parsers.
- Automatic labels:
service_name (from container/OTel), detected_level (when discover_log_levels: true).
vector(0) fallback in alerting prevents "no data" flapping on sparse logs.
[7][METRIC_PATTERNS]
Dictum: Metric queries are pre-aggregated -- prefer for dashboards and alerts.
| [INDEX] |
[PATTERN] |
[QUERY] |
| [1] |
Rate |
rate({job="app"} |= "error" [5m]) |
| [2] |
Count by label |
sum by (app) (count_over_time({ns="prod"} | json [5m])) |
| [3] |
Error percentage |
sum(rate({app="api"} | json | level="error" [5m])) / sum(rate({app="api"}[5m])) * 100 |
| [4] |
Latency P95 |
quantile_over_time(0.95, {app="api"} | json | unwrap duration [5m]) |
| [5] |
Approx Top 10 |
approx_topk(10, sum by (endpoint) (rate({app="api"}[5m]))) |
| [6] |
Dead service |
absent_over_time({app="api"}[5m]) |
| [7] |
Offset compare |
sum(rate(...[5m])) - sum(rate(...[5m] offset 1d)) |
[8][NON-EXISTENT_FEATURES]
Dictum: Generating non-existent operators wastes user time.
| [INDEX] |
[FEATURE] |
[REALITY] |
| [1] |
| dedup |
UI-level in Grafana Explore. Use sum by (field) for dedup. |
| [2] |
| distinct |
Reverted PR #8662. Use count(count by (field) (...)). |
| [3] |
| limit N |
API param &limit=100, Grafana "Line limit", logcli --limit. |
[9][ALLOY_PIPELINE_STAGES]
Dictum: Alloy pipeline stages shape labels and metadata arriving in Loki.
| [INDEX] |
[STAGE] |
[PURPOSE] |
| [1] |
loki.relabel |
Map K8s labels to Loki labels. |
| [2] |
loki.process/multiline |
Aggregate multi-line logs (stack traces). |
| [3] |
loki.process/sampling |
Reduce high-volume streams (cost control). |
[REFERENCE] infrastructure/src/deploy.ts lines 29-36.
[10][RESOURCES]
Dictum: Reference files provide copy-paste patterns and performance guidance.
examples/log_queries.logql -- log parsing, filtering, structured metadata, template functions.examples/metric_queries.logql -- aggregation, alerting, rate/counter functions, label operations.references/best_practices.md -- performance, anti-patterns, recording rules.- context7 MCP (
grafana loki) -- authoritative docs for unclear syntax.
1---2name: logql-generator3description: Generates LogQL queries for Grafana Loki 3.0-3.6 log aggregation and metric extraction. Use when building Loki queries, dashboards, alerts, or optimizing pipeline performance with bloom filters, structured metadata, pattern matching, and approx_topk aggregations.4---5
6# [H1][LOGQL-GENERATOR]
7>**Dictum:** *Pipeline order determines query performance.*
8
9<br>
10
11Generate LogQL for Loki 3.0-3.6 (bloom filters, structured metadata, `approx_topk`). Cross-references: **observability-stack** for Loki deployment, **promql-generator** for PromQL metric queries.
12
13**Tasks:**
141. **Goal** -- Error analysis, performance, security, debugging? Dashboard, alert, ad-hoc?
152. **Sources** -- Labels (`job`, `namespace`, `app`), log format (JSON/logfmt/plain), time range.
163. **Query type** -- Log query (return lines) or metric query (calculate values)?
174. **Plan** -- Plain-English plan, confirm with user before generating.
185. **Generate** -- Apply patterns below; consult `references/best_practices.md`.
196. **Deliver** -- Final query + explanation + usage context (Grafana panel, alert rule, logcli, HTTP API).
20
21---
22## [1][VERSION_MATRIX]
23>**Dictum:** *Version awareness prevents deprecated patterns.*
24
25<br>
26
27| [INDEX] | [VERSION] | [KEY_CHANGES] |
28| :-----: | --------- | --------------------------------------------------------------------------------- |
29| [1] | **3.0** | Bloom filters, structured metadata, pattern match `\|>` / `!>`. |
30| [2] | **3.3** | Bloom acceleration for structured metadata filters. |
31| [3] | **3.5** | Promtail deprecated (EOL March 2, 2026 -- use Alloy), SSD mode deprecated. |
32| [4] | **3.6** | `approx_topk` on querier, reduced JSON/logfmt parser allocations, Loki UI plugin. |
33
34**Guidance:**
35- Promtail EOL March 2, 2026 -- migrate to Alloy.
36- BoltDB deprecated -- use TSDB with v13 schema.
37
38---
39## [2][PIPELINE_ORDER]
40>**Dictum:** *Each stage filters BEFORE the next -- moving expensive operations earlier wastes resources.*
41
42<br>
43
44```
45{stream} -> line filter -> decolorize -> struct metadata -> parser -> label filter -> keep/drop -> format -> aggregate
46 cheapest most expensive
47```
48
49**Guidance:**
50- Structured metadata filters BEFORE parsers enables bloom filter acceleration (3.3+).
51- Line filters (`|=`, `!=`) are O(1) substring checks -- place before parsers.
52- Pattern match `|>` / `!>` uses `<_>` wildcards, 10x faster than regex (3.0+).
53
54**Best-Practices:**
55- Extract only needed JSON fields: `| json level, status` not `| json` -- reduces allocations (3.6).
56- `| decolorize` before `| logfmt` -- ANSI codes break key=value parsing.
57- `| drop instance, pod` before aggregation -- reduces series cardinality.
58
59---
60## [3][STREAM_AND_FILTERS]
61>**Dictum:** *Specific stream selectors minimize chunk scanning.*
62
63<br>
64
65| [INDEX] | [OPERATOR] | [MEANING] | [EXAMPLE] |
66| :-----: | ---------- | ------------------ | --------------------------------------- |
67| [1] | **`\|=`** | Contains. | `{job="app"} \|= "error"` |
68| [2] | **`!=`** | Not contains. | `{job="app"} != "debug"` |
69| [3] | **`\|~`** | Regex match. | `{job="app"} \|~ "error\|fatal"` |
70| [4] | **`!~`** | Regex not match. | `{job="app"} !~ "health\|metrics"` |
71| [5] | **`\|>`** | Pattern match. | `{app="api"} \|> "<_> level=error <_>"` |
72| [6] | **`!>`** | Pattern not match. | `{app="api"} !> "<_> level=debug <_>"` |
73
74---
75## [4][PARSERS]
76>**Dictum:** *Parser selection affects per-line cost.*
77
78<br>
79
80| [INDEX] | [PARSER] | [SYNTAX] | [USE_WHEN] |
81| :-----: | ------------- | --------------------------------------------- | -------------------------------- |
82| [1] | **`pattern`** | `\| pattern "<ip> - <_> <status>"` | Fixed-delimiter structured text. |
83| [2] | **`logfmt`** | `\| logfmt [--strict] [--keep-empty]` | `key=value` pairs. |
84| [3] | **`json`** | `\| json` or `\| json status="response.code"` | JSON (specify fields for perf). |
85| [4] | **`regexp`** | `\| regexp "(?P<field>\\w+)"` | Complex extraction, last resort. |
86| [5] | **`unpack`** | `\| unpack` | Packed JSON from Alloy/Promtail. |
87
88---
89## [5][AGGREGATIONS]
90>**Dictum:** *Metric queries aggregate log entries into time series.*
91
92<br>
93
94### [5.1][LOG_RANGE]
95
96| [INDEX] | [FUNCTION] | [USE_WHEN] |
97| :-----: | --------------------------------- | ---------------------- |
98| [1] | **`rate(log-range)`** | Dashboard rate panels. |
99| [2] | **`count_over_time(log-range)`** | Counting occurrences. |
100| [3] | **`bytes_rate(log-range)`** | Bandwidth monitoring. |
101| [4] | **`absent_over_time(log-range)`** | Dead service alerting. |
102
103---
104### [5.2][UNWRAPPED_RANGE]
105
106| [INDEX] | [FUNCTION] | [USE_WHEN] |
107| :-----: | ------------------------------------ | ------------------------------ |
108| [1] | **`sum/avg/max/min_over_time`** | Aggregate numeric values. |
109| [2] | **`quantile_over_time(phi, range)`** | Latency percentiles from logs. |
110| [3] | **`first/last_over_time`** | Boundary values. |
111| [4] | **`rate_counter(range)`** | Counter-like log values. |
112
113Operators: `sum`, `avg`, `min`, `max`, `count`, `stddev`, `topk`, `bottomk`, `approx_topk`, `sort`, `sort_desc`.
114Grouping: `sum by (label1, label2) (...)` or `sum without (label1) (...)`.
115
116---
117### [5.3][APPROX_TOPK]
118
119`approx_topk(k, expr)` -- probabilistic top-K via count-min sketch, instant queries only. Requires `limits_config.shard_aggregations: [approx_topk]` on querier.
120
121---
122## [6][STRUCTURED_METADATA]
123>**Dictum:** *Structured metadata enables high-cardinality filtering without index impact.*
124
125<br>
126
127Filter AFTER stream selector, BEFORE parsers for bloom acceleration. NOT indexed -- no cardinality impact.
128
129```logql
130{app="api"} | trace_id="abc123" | json | level="error" # CORRECT -- bloom accelerated
131# WRONG: {app="api", trace_id="abc123"} # trace_id is NOT an indexed label
132```
133
134**Guidance:**
135- Bloom filters (3.3+) provide O(1) chunk skipping for string equality and OR filters placed before parsers.
136- Automatic labels: `service_name` (from container/OTel), `detected_level` (when `discover_log_levels: true`).
137- `vector(0)` fallback in alerting prevents "no data" flapping on sparse logs.
138
139---
140## [7][METRIC_PATTERNS]
141>**Dictum:** *Metric queries are pre-aggregated -- prefer for dashboards and alerts.*
142
143<br>
144
145| [INDEX] | [PATTERN] | [QUERY] |
146| :-----: | -------------------- | ----------------------------------------------------------------------------------------- |
147| [1] | **Rate** | `rate({job="app"} \|= "error" [5m])` |
148| [2] | **Count by label** | `sum by (app) (count_over_time({ns="prod"} \| json [5m]))` |
149| [3] | **Error percentage** | `sum(rate({app="api"} \| json \| level="error" [5m])) / sum(rate({app="api"}[5m])) * 100` |
150| [4] | **Latency P95** | `quantile_over_time(0.95, {app="api"} \| json \| unwrap duration [5m])` |
151| [5] | **Approx Top 10** | `approx_topk(10, sum by (endpoint) (rate({app="api"}[5m])))` |
152| [6] | **Dead service** | `absent_over_time({app="api"}[5m])` |
153| [7] | **Offset compare** | `sum(rate(...[5m])) - sum(rate(...[5m] offset 1d))` |
154
155---
156## [8][NON-EXISTENT_FEATURES]
157>**Dictum:** *Generating non-existent operators wastes user time.*
158
159<br>
160
161| [INDEX] | [FEATURE] | [REALITY] |
162| :-----: | ----------------- | --------------------------------------------------------------- |
163| [1] | **`\| dedup`** | UI-level in Grafana Explore. Use `sum by (field)` for dedup. |
164| [2] | **`\| distinct`** | Reverted PR #8662. Use `count(count by (field) (...))`. |
165| [3] | **`\| limit N`** | API param `&limit=100`, Grafana "Line limit", logcli `--limit`. |
166
167---
168## [9][ALLOY_PIPELINE_STAGES]
169>**Dictum:** *Alloy pipeline stages shape labels and metadata arriving in Loki.*
170
171<br>
172
173| [INDEX] | [STAGE] | [PURPOSE] |
174| :-----: | ---------------------------- | ------------------------------------------ |
175| [1] | **`loki.relabel`** | Map K8s labels to Loki labels. |
176| [2] | **`loki.process/multiline`** | Aggregate multi-line logs (stack traces). |
177| [3] | **`loki.process/sampling`** | Reduce high-volume streams (cost control). |
178
179[REFERENCE] `infrastructure/src/deploy.ts` lines 29-36.
180
181---
182## [10][RESOURCES]
183>**Dictum:** *Reference files provide copy-paste patterns and performance guidance.*
184
185<br>
186
187- `examples/log_queries.logql` -- log parsing, filtering, structured metadata, template functions.
188- `examples/metric_queries.logql` -- aggregation, alerting, rate/counter functions, label operations.
189- `references/best_practices.md` -- performance, anti-patterns, recording rules.
190- context7 MCP (`grafana loki`) -- authoritative docs for unclear syntax.
majiayu000