# Stream Aggregation Helper > Design VictoriaMetrics stream aggregation rules to reduce cardinality, sample frequency, or query load. Use whenever the user wants to apply, plan, or troubleshoot stream aggregation (`-streamAggr.config`, `-remoteWrite.streamAggr.config`), asks how to aggregate a high-cardinality metric, mentions downsampling at vmagent, replacing recording rules with stream aggregation, picking the right aggregation output (`total`, `rate_sum`, `sum_samples`, `quantiles`, `histogram_bucket`, etc.), choosing an aggregation interval, deciding where to place vmagent in the delivery pipeline, or whether to scale vmagent for aggregation. Also triggers on "aggregate histograms", "drop labels via vmagent", "reduce ingested samples", "pre-aggregate at vmagent", "stream aggregation accuracy", and "stream aggregation duplicates in cluster". - Skill: `victoriametrics/stream-aggregation-helper` (Agent Skill) - Install (CLI): `npx skillmds@latest add victoriametrics/stream-aggregation-helper` - Raw SKILL.md: https://api.skillmd.com/api/skills/victoriametrics/stream-aggregation-helper/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: DevOps & Infra - Author: victoriametrics (https://skillmd.com/u/victoriametrics) - Updated: 2026-09-09 - Page: https://skillmd.com/skills/victoriametrics/stream-aggregation-helper --- # Stream Aggregation Helper Design, deploy, and verify [stream aggregation](https://docs.victoriametrics.com/victoriametrics/stream-aggregation/) rules for VictoriaMetrics. Stream aggregation reduces cardinality, sample frequency, or query cost by aggregating samples **by time and by labels** at `vmagent` (or single-node vmsingle) **before** they reach storage. The skill walks through five phases: 1. **Gate** — is stream aggregation the right tool for this problem? 2. **Intake** — current cardinality, collection topology, what consumers need. 3. **Decide** — where in the pipeline, interval, output function, `by`/`without`, scaling. 4. **Config** — produce a ready-to-apply YAML plus a phased rollout plan. 5. **Verify** — queries and `vm_streamaggr_*` metrics that confirm the rule is doing what was intended. ## Environment Uses the same env vars as the `victoriametrics-query` skill. The skill works **without** these (interview-only mode) but is much more accurate when it can query the user's VictoriaMetrics directly. ```bash # $VM_METRICS_URL - base URL (optional) # cluster: export VM_METRICS_URL="https://vmselect.example.com/select/0/prometheus" # single: export VM_METRICS_URL="http://localhost:8428" # $VM_CURL_CONFIG - curl config file with auth header (unset if no auth) # Remote: export VM_CURL_CONFIG="$HOME/.config/victoriametrics/curl.conf" # Local: leave unset (defaults to /dev/null - no auth) ``` --- ## Phase 1: Is stream aggregation the right tool? Stream aggregation is the right answer when **all** of these hold: - For every output series that the rule will produce, **all** input samples that contribute to it land on a single aggregator process. This holds in three common topologies: 1. A single `vmagent` (or vmsingle) sees every sample for the matched series. 2. Multiple vmagents shard traffic by some label (e.g., `instance`, `kubernetes_namespace`, `__name__`) and the aggregation **preserves that shard key** — see the shard-key invariant in Phase 3a (case F). Tiered topologies like *tier-1 vmagents shard by `instance` → tier-2 vmagents aggregate within `instance`* are explicitly supported. 3. HA replicas all see every sample **and** the aggregation output is dedup-safe (gauge-style: `min`, `max`, `avg`, `rate_*`, `increase*`, `last`, `quantiles`, etc.). Counter-style outputs (`total`, `total_prometheus`, `histogram_bucket`) cannot be HA-aggregated because each replica maintains independent monotonic state. See the High Availability section. - The user is willing to lose per-source resolution (per-pod, per-instance) for the matched metric, OR only wants to drop a high-cardinality label they don't actually consume. - Consumers (dashboards, alerts, recording rules) either don't need the dropped dimensions, or can be migrated to the aggregated metric name. - State loss on `vmagent` restart is acceptable. Aggregation state is in-memory; on restart the first one or two intervals are discarded by default. **Shard-key invariant**: an output series is correct only if every input sample for it goes to the same aggregator. Translated to rules: the labels used to shard the stream must appear in `by:` (or must not appear in `without:`) of every aggregation rule downstream of the shard. Drop them and you get partial aggregates per shard — silently wrong. If any of these fail, redirect the user: | Situation | Better tool | |-----------|-------------| | Need to drop entire unused metrics | `metric_relabel_configs` with `action: drop` — see `victoriametrics-unused-metrics-analysis` | | Need to drop labels without time-bucketing | `metric_relabel_configs` with `action: labeldrop`, or `-streamAggr.dropInputLabels` | | Need historical downsampling on existing data | vmstorage downsampling (`-downsampling.period`) | | Want a derived metric for queries, not ingestion | Recording rules in vmalert | | Single-node setup but stream-aggregating before storage | `-streamAggr.config` on vmsingle is fine — same engine; just say so | | Behind a load balancer that splits the stream (round-robin) | Shard traffic deterministically by labels (`-remoteWrite.shardByURL.labels`), then aggregate while preserving the shard key. Round-robin LBs split a single output series across replicas → incomplete results. See [Common Mistakes › Behind load balancer](https://docs.victoriametrics.com/victoriametrics/stream-aggregation/#put-aggregator-behind-load-balancer) | | Tiered vmagents (tier-1 shards, tier-2 aggregates) | Supported and common at scale. Aggregation rules on tier-2 must **keep the shard key** in `by:` (or omit it from `without:`). See case F below. | If the gate passes, continue. If not, recommend the alternative and stop. --- ## Phase 2: Intake Collect three groups of facts. Run the live queries in parallel **only if** `VM_METRICS_URL` is set; otherwise ask the user the same questions directly. ### 2a. Cardinality picture What metric/set is the problem? How many series, driven by which labels? ```bash # Top series by metric name (find the offenders) curl -q --config "${VM_CURL_CONFIG:-/dev/null}" -s \ "$VM_METRICS_URL/api/v1/status/tsdb?topN=20" | \ jq '.data | {totalSeries, seriesCountByMetricName, seriesCountByLabelName}' # For each suspect metric, see unique values per label curl -q --config "${VM_CURL_CONFIG:-/dev/null}" -s \ "$VM_METRICS_URL/api/v1/status/tsdb?topN=20&match[]=METRIC_NAME" | \ jq '.data.labelValueCountByLabelName' ``` **Measure post-aggregation cardinality directly.** Once you have a candidate `by:` or `without:`, run the equivalent grouping against current data instead of multiplying per-label unique-value counts. The product-of-labels formula assumes every Cartesian combination of label values exists, which is rarely true — and worse, can be **over-optimistic** when there are correlated labels or labels you forgot to enumerate. The measured count is authoritative: ```bash # Post-aggregation series count, using the labels you plan to KEEP. # Replace and . For counters use rate(...[5m]); for gauges use last_over_time. curl -q --config "${VM_CURL_CONFIG:-/dev/null}" -s \ --data-urlencode 'query=count(sum(rate([5m])) by ())' \ "$VM_METRICS_URL/api/v1/query" | jq '.data.result' # Equivalent form using the labels you plan to DROP: curl -q --config "${VM_CURL_CONFIG:-/dev/null}" -s \ --data-urlencode 'query=count(sum(rate([5m])) without ())' \ "$VM_METRICS_URL/api/v1/query" | jq '.data.result' ``` This gives the actual number of output series the rule will produce *on the current data*, which is the right basis for the savings estimate in Phase 4d. Run it for each candidate rule before committing to a config. Interview questions if live queries are unavailable: - Which metric or metric pattern is the problem? - Approximate total series count for it? - Which labels exist on it, and roughly how many unique values for each? - Counter, gauge, or histogram? - Scrape interval / push frequency? When live queries aren't possible, treat the product-of-labels formula as an **upper bound** only, and note explicitly to the user that the real post-aggregation count must be verified once data is reachable. ### 2b. Collection topology How do samples flow today? Sketch one of these: ``` A: app -> vmagent -> remote write -> TimeSeries DB (vmagent already in path) B: app -> remote write -> TimeSeries DB (no vmagent — must insert one) C: app -> [Prometheus | OTel | StatsD-exporter] -> remote write -> TimeSeries DB D: app -> load balancer -> N vmagents -> TimeSeries DB (LB splits stream — dangerous for SA) E: app -> vmagent (HA pair / N replicas) -> TimeSeries DB (multi-vmagent — dedup needed) F: app -> vmagent cluster (sharded) -> TimeSeries DB (sharded by `-remoteWrite.shardByURL`) ``` The destination is shown as a generic "TimeSeries DB" — stream aggregation is a vmagent-side feature and works with any remote-write-compatible backend (vmstorage/vmsingle, Prometheus, Cortex, Mimir, Thanos receive, etc.). Caveats below that reference vmstorage dedup (`-dedup.minScrapeInterval`) apply specifically when the backend is VictoriaMetrics; for other backends, replicate dedup at the backend or accept that HA aggregation will write duplicates. Interview questions: - Is `vmagent` already in the path? If yes, how many replicas? - Are samples behind a load balancer before reaching the aggregator? - Is the same series fed by exactly one source instance, or by many (e.g., HA pairs, multi-region)? - For K8s users: `VMAgent` CRD (vm-operator) or raw deployment? ### 2c. Consumer intent What do downstream queries actually need? - What resolution do dashboards/alerts query at? (e.g., `rate(...[5m])`, p99 over 5m) - Do any consumers need per-pod / per-instance resolution for this metric? If "no, only service-wide" — that's the strongest signal that SA fits. - Is the metric name referenced in alerting rules or recording rules? Renaming the metric after SA breaks those — track which queries must be updated. - Are there separate destinations that want different aggregations? If yes, use **per-remote-write** configs (`-remoteWrite.streamAggr.config`); otherwise prefer global (`-streamAggr.config`). --- ## Phase 3: Decisions For each metric or group, decide five things. ### 3a. Where to deploy vmagent | Topology | Fits SA? | Notes | |----------|----------|-------| | **A. vmagent already in path, single replica** | Best. | Aggregation is exact — one vmagent sees every sample. | | **B. No vmagent yet** | Insert vmagent **just before** the storage hop. | Adds one process, but you get aggregation + buffering + retries for free. | | **C. Behind Prometheus/OTel collector** | Insert vmagent **between** the collector and remote write, or replace the collector with vmagent if scraping is acceptable. | Don't aggregate inside the collector unless it supports streaming aggregation. | | **D. Behind a load balancer** | **Do not aggregate.** Each vmagent sees a fraction of samples → incomplete aggregates. | Shard traffic by `__name__` or labels first (`-remoteWrite.shardByURL.labels`), or terminate at a single aggregator. | | **E. Multiple vmagent replicas (HA)** | Fits **only** for dedup-safe outputs (gauge-style: `min`, `max`, `avg`, `rate_*`, `increase*`, `last`, `quantiles`, `sum_samples`, etc.). All replicas must receive **identical** input, run **identical** config, and produce **identical** output labels — vmstorage dedup (`-dedup.minScrapeInterval`) collapses the duplicates. Do **not** tag replicas distinctly: that turns dedup-safe series into permanently doubled series at storage. Counter-style outputs (`total`, `total_prometheus`, `histogram_bucket`) cannot be HA'd this way — each replica's monotonic state diverges after restart. | See the High Availability section. | | **F. Sharded vmagent fleet (single tier or tier-1 of a multi-tier topology)** | Fits if the aggregation respects the shard key. The shard label(s) — e.g., `instance`, `kubernetes_namespace`, `__name__` — must appear in every downstream rule's `by:` list, or equivalently must be **absent** from `without:`. Drop the shard key and each shard produces its own slice of the answer → silently wrong totals. | Common configurations: shard by `instance` and aggregate `by: [instance, method, status_code, le]`; shard by `kubernetes_namespace` and aggregate `by: [kubernetes_namespace, workload]`. If the shard key actively splits a series across shards (e.g., sharding by `pod` when you want to aggregate across pods), the topology can't satisfy the shard-key invariant — re-shard, or aggregate on the tier-1 vmagents before sharding. **Never shard by `le` (or `vmrange`) when aggregating histograms** — a single histogram's buckets must all land on the same aggregator. Sharding by `le` routes `le="0.1"`, `le="0.5"`, `le="+Inf"`, etc. for the same histogram to different shards; each shard sees only part of the bucket set, `histogram_quantile()` becomes incoherent, and quantiles silently drift. Shard histograms by a non-bucket label such as `instance`, `kubernetes_namespace`, or `__name__`. | ### 3b. Aggregation interval Two constraints decide `interval`: 1. **Lower bound**: at least **2× the scrape/push interval** of the input. Anything less risks empty buckets when a single sample lands. Default to `2× scrape_interval` (so 30s for 15s scrapes). 2. **Upper bound**: no longer than the **shortest consumer query range**. If alerts use `rate(...[2m])` the interval must be ≤ 1m, ideally 30s. For dashboards with 5m windows, 1m is fine. Other modifiers: - **High delay / sample skew** → use a longer interval, or enable `enable_windows: true` (doubles memory but uses two-window state for accuracy). Critical for histograms. - **Replacing a recording rule** → match the rule's evaluation interval (commonly 1m). - **Downsampling for storage cost** → 1m–5m is typical; longer than 5m sacrifices interactive querying. Default: **`interval: 30s`** for high-frequency app metrics, **`interval: 1m`** when replacing recording rules, **`interval: 5m`** for pure downsampling. ### 3c. Output function This is the most-frequently-wrong choice. Pick by metric kind first, then by what the consumer needs. | Input kind | Consumer needs | Output | Notes | |-----------|----------------|--------|-------| | **Counter** | Sum of rate per output series (e.g., `sum(rate(req_total[5m]))`) | `rate_sum` | Result is a gauge of per-second rate. Query directly with `sum/avg_over_time` — do **not** take `rate()` again. | | **Counter** | Increase per interval, as a counter for `rate()` later | `total` | Output is a monotonic counter resilient to per-source resets and series churn. Query with `rate()`. Use `total_prometheus` to skip the first sample per series. | | **Counter** | Increase per interval, as a value (not a counter) | `increase` / `increase_prometheus` | Output is a gauge of the per-interval delta. | | **Counter** | Average rate across input series | `rate_avg` | Use when you want mean-of-rates instead of sum-of-rates. | | **Histogram bucket** (`*_bucket`, le-style) | `histogram_quantile()` | **`rate_sum`** + `enable_windows: true` | Recommended pattern. Bucket sets must match across aggregated series. | | **Histogram bucket** | Total bucket counts as counter | `total` | Acceptable but more reset-sensitive than `rate_sum`. | | **Gauge** | Latest value | `last` | Cheapest. Matches `last_over_time`. | | **Gauge** | Sum across instances | `sum_samples` | E.g., total memory across pods. | | **Gauge** | Min / Max / Avg | `min` / `max` / `avg` | Same name as the function. | | **Gauge** | Distribution / percentiles | `quantiles(0.5, 0.9, 0.99)` or `histogram_bucket` | `quantiles` emits one output series per phi tagged with a `quantile=""` label. `histogram_bucket` returns VictoriaMetrics-style buckets you can later quantile-query — choose it when you need arbitrary quantiles later, `quantiles(...)` when the set is fixed. | | **Gauge** | Count of unique values | `unique_samples` | Niche; for cardinality-style measurements. | | **Either** | Sample count | `count_samples` / `count_series` | `count_samples` = how many points landed. `count_series` = how many distinct series existed in the interval. | **The single most common mistake** is using `total` on a gauge or `sum_samples` on a counter — both produce nonsense. **Counter → `total`/`rate_*`/`increase*`. Gauge → everything else.** ### 3d. `by` vs `without` - Use **`without:`** when you know the labels to drop (e.g., `pod`, `instance`). The skill should prefer `without` — it's less error-prone if new labels appear later. - Use **`by:`** when you know the exact set of labels to keep, and want a hard guarantee that any new label is dropped. Required when the input has many uncontrolled labels. - Always keep **`le`** when aggregating histogram buckets. - Always keep the labels your downstream queries already group by. - Drop labels that are obviously high-cardinality and not consumed: `pod`, `instance`, `replica`, `container_id`. For **sharded** topologies (case F), keep the shard key. For **HA** topologies (case E), keep label sets *identical* across replicas — no per-replica tag. - **`drop_input_labels` vs `without`/`by`**: both remove a label from the output. Prefer `drop_input_labels` (or the `-streamAggr.dropInputLabels` flag) when the label is **unbounded** (e.g., `request_id`, `trace_id`, `session_id`): it strips the label *before* dedup and state keying, so the aggregator never allocates per-value state for it — even transiently. Use `without`/`by` when the label has a reasonable cardinality but you don't want it on the output. ### 3e. Scale vmagent? A single vmagent handles **millions of samples/sec** on modest hardware. Scale only when: - CPU usage on the aggregating vmagent stays >70% with current load. - `histogram_quantile(0.99, sum(rate(vm_streamaggr_flush_duration_seconds_bucket[1m])) by (vmrange))` approaches the aggregation interval. - Memory pressure from many active output series (each output series ≈ small constant of RAM; with `enable_windows: true` it doubles). - Input rate is sharded across many sources and consolidating in one vmagent is impractical. When scaling: - **Shard by series** (`-remoteWrite.shardByURL.labels`) so each input series lands on exactly one vmagent. Each shard produces a distinct slice of the output (different label values for the shard key); roll-up across shards happens at query time. See Phase 3a (case F). - **HA replicas** (every replica sees every sample): use identical config and identical output labels; vmstorage dedup collapses duplicates. Only valid for dedup-safe outputs. See the High Availability section. ### 3f. How the aggregated output will be consumed Two related decisions: what flows downstream alongside the aggregate, and what the aggregated metric is called. **Input retention** — what happens to the raw (unaggregated) samples after a rule matches them. These are **command-line flags** (or per-remote-write equivalents), not YAML fields: | User intent | Flag setting | Result | |-------------|--------------|--------| | **Aggregates only — drop matched raw** *(default steady state)* | neither flag | Aggregates written. Matched raw dropped. Unmatched-by-rule samples still written. Most common end state. | | **Both raw and aggregates** *(rollout stage 2, or transition periods where consumers still need full resolution)* | `-streamAggr.keepInput=true` (or `-remoteWrite.streamAggr.keepInput=true`) | Aggregates + **all** raw samples written. Cardinality reduction is not yet realized — use temporarily. | | **Aggregates only — drop everything that doesn't match the rules either** *(dedicated archive / long-term path)* | `-streamAggr.dropInput=true` (or `-remoteWrite.streamAggr.dropInput=true`) | Aggregates written. Matched and unmatched raw both dropped. Reach for this only when the destination should contain **nothing but aggregates** — e.g., a per-remote-write SA config aimed at a long-term store, with another (non-aggregating) remote-write carrying full-resolution data elsewhere. | These flags compose with the rollout phases in 4c: typically start with `keepInput=true` during verification, then remove it to reach the default steady state; only set `dropInput=true` if the destination is purely an aggregate sink. **Output metric naming** — by default the aggregated metric name is auto-generated and self-describing: ``` :[_by_][_without_]_ ``` with label lists sorted alphabetically, and a `quantile=""` label per output series when the output is `quantiles(...)`. **Prefer the default** — it makes the aggregate distinguishable from the raw metric at query time and prevents collisions if raw is still flowing. Override only when there's a concrete reason: - `keep_metric_names: true` (per-rule) — keeps the original metric name on the aggregated series. Valid **only** with a single entry in `outputs:`. Safe under the default behavior (matched raw is dropped from remote-write automatically). Risky if you've set `-streamAggr.keepInput=true`, since then aggregates and matched raw share the same name with different semantics — flip back to default before enabling `keep_metric_names`. - `output_relabel_configs:` (per-rule) — full relabeling on the aggregated samples before they're written. Use it to strip the suffix, enforce a project naming convention, or rewrite labels: ```yaml output_relabel_configs: # Strip the auto-added ":interval_..._output" suffix - source_labels: [__name__] regex: '(.+):.+' target_label: __name__ replacement: '$1' # Or enforce an "agg_" prefix - source_labels: [__name__] regex: '(.+)' target_label: __name__ replacement: 'agg_$1' ``` When you rename, plan the corresponding consumer migration (dashboards, alerts, recording rules) — see Phase 4c stage 4. ### 3g. Resolution vs. SA interval — replacing `sum(rate(X[5m]))` A common request is "I have `sum(rate(req_total[5m]))` in my dashboards and alerts — please pre-aggregate it." The naive mapping is to match the SA interval to the rate window: ```yaml - match: 'req_total' interval: 5m # too coarse — see below by: [] outputs: [rate_sum] ``` This produces **one datapoint every 5 minutes**. That resolution is often too coarse for alerting (alerts evaluate every 30s–1m and need recent samples) and for graphs (5-minute step looks like a staircase). The original `rate(...[5m])` also gave you a *rolling* 5m window evaluated at every scrape; a 5m SA interval gives you a tumbling 5m bucket — those are not the same. **Recommended default**: use a finer SA interval (typically `1m`) and reconstruct the rolling smoothing at **query time** with `avg_over_time`: ```yaml - match: 'req_total' interval: 1m by: [] outputs: [rate_sum] ``` Query rewrites: ```promql # Original sum(rate(req_total[5m])) # After SA with interval: 1m, rate_sum (rate_sum is already per-second, so don't rate() again) sum(avg_over_time(req_total:1m_..._rate_sum[5m])) # Original with grouping sum by (route) (rate(req_total[5m])) # After SA — use a subquery to apply avg_over_time after the sum-by avg_over_time( (sum(req_total:1m_..._rate_sum) by (route))[5m:1m] ) ``` The `[5m:1m]` subquery means "evaluate the inner expression at 1m steps over a 5m window" — i.e., exactly the rolling-5m-window-every-1m behavior the original `rate(...[5m])` produced. Trade-off table: | SA `interval` | Samples written per 5m | When it fits | |---------------|------------------------|--------------| | `5m` | 1 | Backfill / long-term archive only. Acceptable when no consumer queries faster than 5m and a staircase graph is fine. | | `1m` *(default for `[5m]` consumers)* | 5 | High-resolution graphs and alerts; `avg_over_time(...[5m])` at query time reproduces the original smoothing. | | `30s` | 10 | Consumers use shorter windows (`[2m]`, `[1m]`) or zoom into sub-minute graphs. Often overkill. | The same principle applies to `increase`, `increase_prometheus`, `rate_avg`, and gauge outputs (`min`, `max`, `avg`, `sum_samples`): pick the SA interval based on the **fastest cadence any consumer wants to query at**, not the largest range window the consumer uses. Reconstruct the range window with the matching `*_over_time` wrapper at query time. --- ## Phase 4: Config + Rollout Produce a single, self-contained plan that the user can read top-to-bottom and apply without clicking through to docs. The expected output shape — adapt to the actual rule, omit sections that don't apply: > **Stream aggregation plan: `http_duration_p99`** > > | Decision | Choice | Why | > |----------|--------|-----| > | Placement | Existing tier-2 vmagent | Already in path; tier-1 shards by `kubernetes_namespace`, tier-2 aggregates within it | > | Interval | `1m` | 2× scrape (15s); consumers use `[5m]` windows — reconstruct rolling avg at query time (Phase 3g) | > | Output | `rate_sum` + `enable_windows: true` | Histogram → recommended pattern; consumers want p99 | > | Grouping | `without: [pod, instance, path]` | Drop per-source + raw URL; keep `kubernetes_namespace` (shard key), `method`, `status_code`, `route`, `le` | > | Input retention | default (matched raw dropped from remote write after cutover) | No consumer needs raw post-cutover | > | Series before → after | ~5.3M → **~8.8K** (measured via `count(sum by (...)(rate(...[5m])))`) | ~600× reduction | > | HA / sharding | Tier-2 has 2 HA replicas; identical config, identical labels, dedup at storage | Output is `rate_sum` (HA-safe). Do not tag replicas. | > >
> Stream aggregation config — paste into -streamAggr.config (or VMAgent.spec.streamAggrConfig.rules) > > ```yaml > - name: http_duration_p99 > match: 'http_request_duration_seconds_bucket' > interval: 1m > enable_windows: true > without: [pod, instance, path] > outputs: [rate_sum] > ``` >
> >
> Rollout > > 1. Deploy config with `-streamAggr.keepInput=true` (or `streamAggrConfig.keepInput: true` on the CRD). Reload (`kill -SIGHUP …` or `/-/reload`). > 2. Wait ≥1 hour, then run the cross-check query (verification block below). Aggregated p99 should track raw p99 within a few percent. > 3. Remove `-streamAggr.keepInput=true`. Reload. Matched raw is now dropped from remote-write by default — cardinality reduction is realized. > 4. Migrate dashboards / alerts to `http_request_duration_seconds_bucket:1m_without_instance_path_pod_rate_sum`. Suffix labels are sorted alphabetically. >
> >
> Verification queries > > ```promql > # Cross-check: raw vs aggregated p99 (run during stage 2 with keepInput on) > histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))) > > histogram_quantile(0.99, > sum by (le) ( > avg_over_time(http_request_duration_seconds_bucket:1m_without_instance_path_pod_rate_sum[5m]) > ) > ) > > # Watch on vmagent /metrics > # - histogram_quantile(0.99, sum(rate(vm_streamaggr_flush_duration_seconds_bucket{name="http_duration_p99"}[1m])) by (vmrange)) << 1m > # - vm_streamaggr_counter_resets_total — flat (spikes = HA collision) > # - count({__name__="http_request_duration_seconds_bucket:1m_without_instance_path_pod_rate_sum"}) ≈ 8800 > ``` >
Adapt the structure for the actual scenario: a gauge with `quantiles(0.95)` won't need `enable_windows`; a downsampling rule replacing a recording rule may skip the cross-check (no raw to compare to). Drop the HA row if the deployment is single-replica. ### 4a. Generate the config Template: ```yaml - name: # shows up in vm_streamaggr_* metrics match: '' # e.g. '{__name__=~"http_request_.*"}' interval: <30s|1m|5m> # enable_windows: true # REQUIRED for histograms; recommended when samples are delayed # staleness_interval: 2m # default 2*interval; raise it if input is sporadic # ignore_first_sample_interval: 2m # for total/increase/histogram_bucket after restart # ignore_old_samples: true # if the stream contains backfilled / replayed data, is automatically applied when enable_windows are enabled without: [pod, instance, replica] # or: by: [route, method, status_code, le] outputs: [rate_sum] # see output picker above # keep_metric_names: true # only with a single output; drops the ":interval_..." suffix # drop_input_labels: [container_id] # drops labels *before* dedup + aggregation # input_relabel_configs: [...] # rare; transform labels before aggregating # output_relabel_configs: [...] # rare; e.g. strip the auto-added suffix ``` **Do not** put `keep_input` or `drop_input` inside the YAML — those are command-line flags (`-streamAggr.keepInput`, `-streamAggr.dropInput`, plus per-remote-write variants). Several common guides get this wrong. Chose `name` field to reflect original metric name and aggregation output, but keep it simple enough that's easy to understand. ### 4b. Where the config lives | Mechanism | Use when | |-----------|----------| | `-streamAggr.config` | One global config replicated to all `-remoteWrite.url` destinations. **Default choice.** Aggregation runs once. | | `-remoteWrite.streamAggr.config` (per URL) | Different destinations need different aggregates (e.g., long-term store gets aggregated only). Aggregation runs per destination — more CPU. | | vmsingle `-streamAggr.config` | Single-node setup; same engine, same YAML. | ### 4c. Phased rollout 1. **Stage 1 — observe**. Deploy with input retained, so both raw and aggregated land in storage: - Don't set `-streamAggr.dropInput` (default keeps unmatched + writes aggregates). - Set `-streamAggr.keepInput=true` to **also** write the matched raw samples. This lets you compare raw vs aggregated queries side-by-side. 2. **Stage 2 — verify** (see Phase 5). Aggregated output should match the corresponding `sum(rate(...))` / `histogram_quantile(...)` over raw for at least an hour, including across a vmagent restart if possible. 3. **Stage 3 — cut over**. Remove `-streamAggr.keepInput`. That's it — the cardinality reduction is realized at this point. The default SA behavior writes aggregates and **drops matched raw samples** from the remote-write output (only the unmatched-by-rule samples continue to flow as raw). You do **not** need to drop the raw metric at the scrape side; in fact you must **not**, because vmagent's pipeline is `scrape → relabel → dedup → stream aggregation → remote write`, so a scrape-side drop removes the input the aggregator needs and the rule produces nothing. Optionally set `-streamAggr.dropInput=true` only if you also want to drop all *unmatched-by-rule* samples (rare; appropriate when this remote-write destination is dedicated to aggregates and another remote-write carries the full-resolution data). See Phase 3f for the three input-retention modes. 4. **Stage 4 — migrate consumers**. Update dashboards / alerts to the new metric name (with the auto-added suffix). The suffix format is `:[_by_][_without_]_` where **both** label lists are sorted alphabetically — `without: [pod, instance, path]` yields `_without_instance_path_pod_`; `by: [function, region, cold_start]` yields `_by_cold_start_function_region_`. For `outputs: [quantiles(0.5, 0.99)]` each output series also carries a `quantile=""` label. If you'd rather keep the original metric name, set `keep_metric_names: true` (valid only with a single output), and make sure raw is fully dropped or you'll have two series with the same name and different semantics. Reload without restart: ```bash kill -SIGHUP $(pidof vmagent) # or curl -X POST http://vmagent:8429/-/reload ``` ### 4d. Estimate the savings For each rule, give the user two numbers: - **Series before** (current): from `seriesCountByMetricName` in TSDB status, or user-reported. - **Series after** (measured): the count returned by the `count(sum(rate([5m])) by ())` query in Phase 2a, run against current data. This is the authoritative number — the rule will produce roughly this many output series on the current data. Only fall back to the product-of-labels formula when live data isn't reachable, and label it as an upper bound: ``` output_series ≤ Π(unique values of each label in `by`, including __name__) ``` The product overestimates when label values are correlated (most routes don't return every status code) and can also *under*estimate when there are labels the user didn't enumerate — both reasons to prefer the measured count. Sanity check: if the measured after-aggregation count is **larger** than before, the `by`/`without` is wrong — you're keeping labels that produce more unique combinations than expected. --- ## Phase 5: Verify After the rule is live, run these checks. Use them in stage 2 (with raw retained) and again in stage 3. ### 5a. Stream aggregation internal metrics Check metrics exposed by vmagent that does the aggregation (or check them on the vmagent [Grafana dashboard](https://grafana.com/grafana/dashboards/12683-victoriametrics-vmagent/) in `Stream aggregation` section): | Metric | What it tells you | |---------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `vm_streamaggr_matched_samples_total{name="..."}` | Input samples matched by the rule. Should match expected scrape rate × series count. | | `vm_streamaggr_output_samples_total{name="..."}` | Aggregated samples emitted. Roughly `output_series / aggregation_interval`. | | `vm_streamaggr_flush_duration_seconds_bucket` | Time spent flushing each interval. Can be calculated as `histogram_quantile(0.99, rate(vm_streamaggr_flush_duration_seconds_bucket[5m]))` to get 99th percentile. Should be well below `interval`. If close to interval, scale vmagent or raise `interval`. | | `vm_streamaggr_ignored_samples_total` | Samples dropped due to different reasons. Spikes in `too_old` reasin may indicates upstream delays. See also `vm_streamaggr_samples_lag_seconds_bucket` for the delivery lag. | | `vm_streamaggr_counter_resets_total` | Counter resets observed. **Sudden spike** = duplicate or colliding input samples; check for upstream issues or a misconfigured topology (e.g., HA replicas with a counter-style output, see High Availability section). | ### 5b. Cross-check aggregated vs raw (stage 2 only) For a counter aggregated with `rate_sum`: ```promql # raw sum by (route, method, le) (rate(http_request_duration_seconds_bucket[5m])) # aggregated (rate_sum is already a per-second rate — sum it, don't rate it again) sum by (route, method, le) (http_request_duration_seconds_bucket:30s_without_pod_instance_rate_sum) ``` The two should track within a few percent. If they diverge sharply, suspect: - Multi-vmagent collision (Phase 3a, case E). - Behind a load balancer (Phase 3a, case D). - Mismatched `dedup_interval` vs storage `-dedup.minScrapeInterval`. - For histograms: sample delays — enable `enable_windows` and/or raise `interval`. For histograms via `rate_sum`: ```promql # raw p99 histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))) # aggregated p99 histogram_quantile(0.99, sum by (le) (avg_over_time(http_request_duration_seconds_bucket:30s_without_pod_instance_rate_sum[5m]))) ``` ### 5c. Sanity checks - Output series count from `count({__name__=""})` ≈ your Phase 4d estimate. - No alerts about `histogram_quantile(0.99, sum(rate(vm_streamaggr_flush_duration_seconds_bucket[1m])) by (vmrange))` approaching interval. - For `total`/`increase` outputs: counter is monotonic; resets only on vmagent restart (which is expected unless `flush_on_shutdown: true` is set). --- ## High Availability Running multiple vmagent replicas for resilience is supported, but only under specific conditions — stream aggregation maintains state in the aggregator process, and that state cannot be replicated across replicas. ### The mechanism Multiple replicas produce the **same** output series — identical label sets — and let vmstorage's deduplication collapse the duplicates. This works **only** when: 1. Every replica receives **identical input** — same scrape targets, or fan-out (not round-robin) remote-write. A round-robin load balancer in front of HA replicas splits the stream, which both defeats HA *and* produces incomplete aggregates per replica. 2. Every replica runs **identical aggregation configuration**. 3. Every replica produces output samples with **identical labels** (no per-replica tagging — see below). 4. vmstorage has `-dedup.minScrapeInterval` set (usually equal to the aggregation `interval`); the per-rule `dedup_interval` should match. 5. The chosen aggregation output is dedup-safe (next section). **Do not** tag HA replicas with distinct labels (`-remoteWrite.label=vmagent=A`, `=B`). Distinct labels make the series genuinely different — vmstorage won't dedup, you pay double the storage, and the consumer must `sum without (vmagent)` everywhere. The "tag each replica" pattern is for *sharded* topologies (case F), where each replica sees a disjoint slice of inputs and must keep them apart, not for HA where replicas see the same inputs. ### HA-safe vs unsafe outputs | Output | HA-safe? | Why | |--------|---------|-----| | `min`, `max`, `avg`, `last`, `sum_samples`, `count_samples`, `count_series`, `unique_samples`, `stddev`, `stdvar`, `quantiles(...)` | ✅ | Each interval's value is computed independently from samples seen in that interval. Two replicas that see the same input converge to the same value → dedup is clean. | | `rate_sum`, `rate_avg`, `increase`, `increase_prometheus` | ✅ (with caveat) | Output is a **per-interval gauge** of rate or delta, not a monotonic counter. Internal per-series state can briefly diverge after one replica restarts (until the next staleness window), but the output values converge. | | `total`, `total_prometheus` | ❌ | Output is a **monotonically increasing counter** held in aggregator memory. After one replica restarts, its output resets to 0 while the other's keeps climbing. Dedup picks one sample per (series, timestamp) — with two divergent counter streams emitting at the same series, the consumer sees an alternating sequence, and `rate()` reads the backward steps as counter resets, producing visible jumps and spikes. There is no clean way to dedupe two divergent monotonic counters. | | `histogram_bucket` | ❌ | Per-bucket counters accumulate the same way `total` does. Same divergence problem. | For consumers that insist on a counter shape (e.g., legacy alerts using `rate(...[5m])`), the HA-safe options are: - Switch the aggregation to `rate_sum` and update the query (`rate_sum` is already a per-second rate — sum/avg it instead of taking `rate()`). - Switch to `increase` and use `increase`-style queries. - Run a single aggregator (active/passive failover) instead of true HA — accept brief gaps during failover rather than divergent monotonic state. ### HA vs sharding — different patterns, don't combine on the same dimension - **Sharded** (case F): each replica sees a *disjoint slice* of inputs. Each emits distinct output series — either by keeping the shard key in `by:` (preferred — roll up at query time), or by tagging with a unique label. Different replicas produce different values intentionally. - **HA** (case E): every replica sees the *same* inputs and emits the *same* output labels. Different replicas produce equal values that …(truncated)