Stream Aggregation Helper

Stream Aggregation Helper

Design, deploy, and verify stream aggregation rules for VictoriaMetrics. Stream aggregation reduces cardinality, sample frequency, or query cost by aggregating samples by time and by labels at vmagent (or single-node vmsingle) before they reach storage.

The skill walks through five phases:

1. Gate — is stream aggregation the right tool for this problem?
2. Intake — current cardinality, collection topology, what consumers need.
3. Decide — where in the pipeline, interval, output function, by/without, scaling.
4. Config — produce a ready-to-apply YAML plus a phased rollout plan.
5. Verify — queries and vm_streamaggr_* metrics that confirm the rule is doing what was intended.

Environment

Uses the same env vars as the victoriametrics-query skill. The skill works without these (interview-only mode) but is much more accurate when it can query the user's VictoriaMetrics directly.

# $VM_METRICS_URL - base URL (optional)
#   cluster: export VM_METRICS_URL="https://vmselect.example.com/select/0/prometheus"
#   single:  export VM_METRICS_URL="http://localhost:8428"
# $VM_CURL_CONFIG - curl config file with auth header (unset if no auth)
#   Remote: export VM_CURL_CONFIG="$HOME/.config/victoriametrics/curl.conf"
#   Local:  leave unset (defaults to /dev/null - no auth)

Phase 1: Is stream aggregation the right tool?

Stream aggregation is the right answer when all of these hold:

• For every output series that the rule will produce, all input samples that contribute to it land on a single aggregator process. This holds in three common topologies:
  1. A single vmagent (or vmsingle) sees every sample for the matched series.
  2. Multiple vmagents shard traffic by some label (e.g., instance, kubernetes_namespace, __name__) and the aggregation preserves that shard key — see the shard-key invariant in Phase 3a (case F). Tiered topologies like tier-1 vmagents shard by instance → tier-2 vmagents aggregate within instance are explicitly supported.
  3. HA replicas all see every sample and the aggregation output is dedup-safe (gauge-style: min, max, avg, rate_*, increase*, last, quantiles, etc.). Counter-style outputs (total, total_prometheus, histogram_bucket) cannot be HA-aggregated because each replica maintains independent monotonic state. See the High Availability section.
• The user is willing to lose per-source resolution (per-pod, per-instance) for the matched metric, OR only wants to drop a high-cardinality label they don't actually consume.
• Consumers (dashboards, alerts, recording rules) either don't need the dropped dimensions, or can be migrated to the aggregated metric name.
• State loss on vmagent restart is acceptable. Aggregation state is in-memory; on restart the first one or two intervals are discarded by default.

Shard-key invariant: an output series is correct only if every input sample for it goes to the same aggregator. Translated to rules: the labels used to shard the stream must appear in by: (or must not appear in without:) of every aggregation rule downstream of the shard. Drop them and you get partial aggregates per shard — silently wrong.

If any of these fail, redirect the user:

If the gate passes, continue. If not, recommend the alternative and stop.

Phase 2: Intake

Collect three groups of facts. Run the live queries in parallel only if VM_METRICS_URL is set; otherwise ask the user the same questions directly.

2a. Cardinality picture

What metric/set is the problem? How many series, driven by which labels?

# Top series by metric name (find the offenders)
curl -q --config "${VM_CURL_CONFIG:-/dev/null}" -s \
  "$VM_METRICS_URL/api/v1/status/tsdb?topN=20" | \
  jq '.data | {totalSeries, seriesCountByMetricName, seriesCountByLabelName}'

# For each suspect metric, see unique values per label
curl -q --config "${VM_CURL_CONFIG:-/dev/null}" -s \
  "$VM_METRICS_URL/api/v1/status/tsdb?topN=20&match[]=METRIC_NAME" | \
  jq '.data.labelValueCountByLabelName'

Measure post-aggregation cardinality directly. Once you have a candidate by: or without:, run the equivalent grouping against current data instead of multiplying per-label unique-value counts. The product-of-labels formula assumes every Cartesian combination of label values exists, which is rarely true — and worse, can be over-optimistic when there are correlated labels or labels you forgot to enumerate. The measured count is authoritative:

# Post-aggregation series count, using the labels you plan to KEEP.
# Replace <metric> and <kept_labels>. For counters use rate(...[5m]); for gauges use last_over_time.
curl -q --config "${VM_CURL_CONFIG:-/dev/null}" -s \
  --data-urlencode 'query=count(sum(rate(<metric>[5m])) by (<kept_labels>))' \
  "$VM_METRICS_URL/api/v1/query" | jq '.data.result'

# Equivalent form using the labels you plan to DROP:
curl -q --config "${VM_CURL_CONFIG:-/dev/null}" -s \
  --data-urlencode 'query=count(sum(rate(<metric>[5m])) without (<dropped_labels>))' \
  "$VM_METRICS_URL/api/v1/query" | jq '.data.result'

This gives the actual number of output series the rule will produce on the current data, which is the right basis for the savings estimate in Phase 4d. Run it for each candidate rule before committing to a config.

Interview questions if live queries are unavailable:

• Which metric or metric pattern is the problem?
• Approximate total series count for it?
• Which labels exist on it, and roughly how many unique values for each?
• Counter, gauge, or histogram?
• Scrape interval / push frequency?

When live queries aren't possible, treat the product-of-labels formula as an upper bound only, and note explicitly to the user that the real post-aggregation count must be verified once data is reachable.

2b. Collection topology

How do samples flow today? Sketch one of these:

A: app  -> vmagent -> remote write -> TimeSeries DB         (vmagent already in path)
B: app  -> remote write -> TimeSeries DB                    (no vmagent — must insert one)
C: app  -> [Prometheus | OTel | StatsD-exporter] -> remote write -> TimeSeries DB
D: app  -> load balancer -> N vmagents -> TimeSeries DB     (LB splits stream — dangerous for SA)
E: app  -> vmagent (HA pair / N replicas) -> TimeSeries DB  (multi-vmagent — dedup needed)
F: app  -> vmagent cluster (sharded) -> TimeSeries DB       (sharded by `-remoteWrite.shardByURL`)

The destination is shown as a generic "TimeSeries DB" — stream aggregation is a vmagent-side feature and works with any remote-write-compatible backend (vmstorage/vmsingle, Prometheus, Cortex, Mimir, Thanos receive, etc.). Caveats below that reference vmstorage dedup (-dedup.minScrapeInterval) apply specifically when the backend is VictoriaMetrics; for other backends, replicate dedup at the backend or accept that HA aggregation will write duplicates.

Interview questions:

• Is vmagent already in the path? If yes, how many replicas?
• Are samples behind a load balancer before reaching the aggregator?
• Is the same series fed by exactly one source instance, or by many (e.g., HA pairs, multi-region)?
• For K8s users: VMAgent CRD (vm-operator) or raw deployment?

2c. Consumer intent

What do downstream queries actually need?

• What resolution do dashboards/alerts query at? (e.g., rate(...[5m]), p99 over 5m)
• Do any consumers need per-pod / per-instance resolution for this metric? If "no, only service-wide" — that's the strongest signal that SA fits.
• Is the metric name referenced in alerting rules or recording rules? Renaming the metric after SA breaks those — track which queries must be updated.
• Are there separate destinations that want different aggregations? If yes, use per-remote-write configs (-remoteWrite.streamAggr.config); otherwise prefer global (-streamAggr.config).

Phase 3: Decisions

For each metric or group, decide five things.

3a. Where to deploy vmagent

3b. Aggregation interval

Two constraints decide interval:

1. Lower bound: at least 2× the scrape/push interval of the input. Anything less risks empty buckets when a single sample lands. Default to 2× scrape_interval (so 30s for 15s scrapes).
2. Upper bound: no longer than the shortest consumer query range. If alerts use rate(...[2m]) the interval must be ≤ 1m, ideally 30s. For dashboards with 5m windows, 1m is fine.

Other modifiers:

• High delay / sample skew → use a longer interval, or enable enable_windows: true (doubles memory but uses two-window state for accuracy). Critical for histograms.
• Replacing a recording rule → match the rule's evaluation interval (commonly 1m).
• Downsampling for storage cost → 1m–5m is typical; longer than 5m sacrifices interactive querying.

Default: interval: 30s for high-frequency app metrics, interval: 1m when replacing recording rules, interval: 5m for pure downsampling.

3c. Output function

This is the most-frequently-wrong choice. Pick by metric kind first, then by what the consumer needs.

The single most common mistake is using total on a gauge or sum_samples on a counter — both produce nonsense. Counter → total/rate_*/increase*. Gauge → everything else.

3d. by vs without

• Use without: when you know the labels to drop (e.g., pod, instance). The skill should prefer without — it's less error-prone if new labels appear later.
• Use by: when you know the exact set of labels to keep, and want a hard guarantee that any new label is dropped. Required when the input has many uncontrolled labels.
• Always keep le when aggregating histogram buckets.
• Always keep the labels your downstream queries already group by.
• Drop labels that are obviously high-cardinality and not consumed: pod, instance, replica, container_id. For sharded topologies (case F), keep the shard key. For HA topologies (case E), keep label sets identical across replicas — no per-replica tag.
• drop_input_labels vs without/by: both remove a label from the output. Prefer drop_input_labels (or the -streamAggr.dropInputLabels flag) when the label is unbounded (e.g., request_id, trace_id, session_id): it strips the label before dedup and state keying, so the aggregator never allocates per-value state for it — even transiently. Use without/by when the label has a reasonable cardinality but you don't want it on the output.

3e. Scale vmagent?

A single vmagent handles millions of samples/sec on modest hardware. Scale only when:

• CPU usage on the aggregating vmagent stays >70% with current load.
• histogram_quantile(0.99, sum(rate(vm_streamaggr_flush_duration_seconds_bucket[1m])) by (vmrange)) approaches the aggregation interval.
• Memory pressure from many active output series (each output series ≈ small constant of RAM; with enable_windows: true it doubles).
• Input rate is sharded across many sources and consolidating in one vmagent is impractical.

When scaling:

• Shard by series (-remoteWrite.shardByURL.labels) so each input series lands on exactly one vmagent. Each shard produces a distinct slice of the output (different label values for the shard key); roll-up across shards happens at query time. See Phase 3a (case F).
• HA replicas (every replica sees every sample): use identical config and identical output labels; vmstorage dedup collapses duplicates. Only valid for dedup-safe outputs. See the High Availability section.

3f. How the aggregated output will be consumed

Two related decisions: what flows downstream alongside the aggregate, and what the aggregated metric is called.

Input retention — what happens to the raw (unaggregated) samples after a rule matches them. These are command-line flags (or per-remote-write equivalents), not YAML fields:

These flags compose with the rollout phases in 4c: typically start with keepInput=true during verification, then remove it to reach the default steady state; only set dropInput=true if the destination is purely an aggregate sink.

Output metric naming — by default the aggregated metric name is auto-generated and self-describing:

<original_name>:<interval>[_by_<labels>][_without_<labels>]_<output>

with label lists sorted alphabetically, and a quantile="<phi>" label per output series when the output is quantiles(...). Prefer the default — it makes the aggregate distinguishable from the raw metric at query time and prevents collisions if raw is still flowing.

Override only when there's a concrete reason:

• keep_metric_names: true (per-rule) — keeps the original metric name on the aggregated series. Valid only with a single entry in outputs:. Safe under the default behavior (matched raw is dropped from remote-write automatically). Risky if you've set -streamAggr.keepInput=true, since then aggregates and matched raw share the same name with different semantics — flip back to default before enabling keep_metric_names.

• output_relabel_configs: (per-rule) — full relabeling on the aggregated samples before they're written. Use it to strip the suffix, enforce a project naming convention, or rewrite labels:

  output_relabel_configs:
    # Strip the auto-added ":interval_..._output" suffix
    - source_labels: [__name__]
      regex: '(.+):.+'
      target_label: __name__
      replacement: '$1'
  
    # Or enforce an "agg_" prefix
    - source_labels: [__name__]
      regex: '(.+)'
      target_label: __name__
      replacement: 'agg_$1'
  

  When you rename, plan the corresponding consumer migration (dashboards, alerts, recording rules) — see Phase 4c stage 4.

3g. Resolution vs. SA interval — replacing sum(rate(X[5m]))

A common request is "I have sum(rate(req_total[5m])) in my dashboards and alerts — please pre-aggregate it." The naive mapping is to match the SA interval to the rate window:

- match: 'req_total'
  interval: 5m            # too coarse — see below
  by: [<labels>]
  outputs: [rate_sum]

This produces one datapoint every 5 minutes. That resolution is often too coarse for alerting (alerts evaluate every 30s–1m and need recent samples) and for graphs (5-minute step looks like a staircase). The original rate(...[5m]) also gave you a rolling 5m window evaluated at every scrape; a 5m SA interval gives you a tumbling 5m bucket — those are not the same.

Recommended default: use a finer SA interval (typically 1m) and reconstruct the rolling smoothing at query time with avg_over_time:

- match: 'req_total'
  interval: 1m
  by: [<labels>]
  outputs: [rate_sum]

Query rewrites:

# Original
sum(rate(req_total[5m]))

# After SA with interval: 1m, rate_sum  (rate_sum is already per-second, so don't rate() again)
sum(avg_over_time(req_total:1m_..._rate_sum[5m]))

# Original with grouping
sum by (route) (rate(req_total[5m]))

# After SA — use a subquery to apply avg_over_time after the sum-by
avg_over_time(
  (sum(req_total:1m_..._rate_sum) by (route))[5m:1m]
)

The [5m:1m] subquery means "evaluate the inner expression at 1m steps over a 5m window" — i.e., exactly the rolling-5m-window-every-1m behavior the original rate(...[5m]) produced.

Trade-off table:

The same principle applies to increase, increase_prometheus, rate_avg, and gauge outputs (min, max, avg, sum_samples): pick the SA interval based on the fastest cadence any consumer wants to query at, not the largest range window the consumer uses. Reconstruct the range window with the matching *_over_time wrapper at query time.

Phase 4: Config + Rollout

Produce a single, self-contained plan that the user can read top-to-bottom and apply without clicking through to docs. The expected output shape — adapt to the actual rule, omit sections that don't apply:

Stream aggregation plan: http_duration_p99

Decision	Choice	Why
Placement	Existing tier-2 vmagent	Already in path; tier-1 shards by kubernetes_namespace, tier-2 aggregates within it
Interval	1m	2× scrape (15s); consumers use [5m] windows — reconstruct rolling avg at query time (Phase 3g)
Output	rate_sum + enable_windows: true	Histogram → recommended pattern; consumers want p99
Grouping	without: [pod, instance, path]	Drop per-source + raw URL; keep kubernetes_namespace (shard key), method, status_code, route, le
Input retention	default (matched raw dropped from remote write after cutover)	No consumer needs raw post-cutover
Series before → after	5.3M → **8.8K** (measured via count(sum by (...)(rate(...[5m]))))	~600× reduction
HA / sharding	Tier-2 has 2 HA replicas; identical config, identical labels, dedup at storage	Output is rate_sum (HA-safe). Do not tag replicas.

- name: http_duration_p99
  match: 'http_request_duration_seconds_bucket'
  interval: 1m
  enable_windows: true
  without: [pod, instance, path]
  outputs: [rate_sum]

1. Deploy config with -streamAggr.keepInput=true (or streamAggrConfig.keepInput: true on the CRD). Reload (kill -SIGHUP … or /-/reload).
2. Wait ≥1 hour, then run the cross-check query (verification block below). Aggregated p99 should track raw p99 within a few percent.
3. Remove -streamAggr.keepInput=true. Reload. Matched raw is now dropped from remote-write by default — cardinality reduction is realized.
4. Migrate dashboards / alerts to http_request_duration_seconds_bucket:1m_without_instance_path_pod_rate_sum. Suffix labels are sorted alphabetically.

# Cross-check: raw vs aggregated p99 (run during stage 2 with keepInput on)
histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

histogram_quantile(0.99,
  sum by (le) (
    avg_over_time(http_request_duration_seconds_bucket:1m_without_instance_path_pod_rate_sum[5m])
  )
)

# Watch on vmagent /metrics
# - histogram_quantile(0.99, sum(rate(vm_streamaggr_flush_duration_seconds_bucket{name="http_duration_p99"}[1m])) by (vmrange)) << 1m
# - vm_streamaggr_counter_resets_total — flat (spikes = HA collision)
# - count({__name__="http_request_duration_seconds_bucket:1m_without_instance_path_pod_rate_sum"}) ≈ 8800

Adapt the structure for the actual scenario: a gauge with quantiles(0.95) won't need enable_windows; a downsampling rule replacing a recording rule may skip the cross-check (no raw to compare to). Drop the HA row if the deployment is single-replica.

4a. Generate the config

Template:

- name: <descriptive_name>             # shows up in vm_streamaggr_* metrics
  match: '<series_selector>'           # e.g. '{__name__=~"http_request_.*"}'
  interval: <30s|1m|5m>
  # enable_windows: true               # REQUIRED for histograms; recommended when samples are delayed
  # staleness_interval: 2m             # default 2*interval; raise it if input is sporadic
  # ignore_first_sample_interval: 2m   # for total/increase/histogram_bucket after restart
  # ignore_old_samples: true           # if the stream contains backfilled / replayed data, is automatically applied when enable_windows are enabled
  without: [pod, instance, replica]    # or: by: [route, method, status_code, le]
  outputs: [rate_sum]                  # see output picker above
  # keep_metric_names: true            # only with a single output; drops the ":interval_..." suffix
  # drop_input_labels: [container_id]  # drops labels *before* dedup + aggregation
  # input_relabel_configs: [...]       # rare; transform labels before aggregating
  # output_relabel_configs: [...]      # rare; e.g. strip the auto-added suffix

Do not put keep_input or drop_input inside the YAML — those are command-line flags (-streamAggr.keepInput, -streamAggr.dropInput, plus per-remote-write variants). Several common guides get this wrong.

Chose name field to reflect original metric name and aggregation output, but keep it simple enough that's easy to understand.

4b. Where the config lives

4c. Phased rollout

1. Stage 1 — observe. Deploy with input retained, so both raw and aggregated land in storage:

   • Don't set -streamAggr.dropInput (default keeps unmatched + writes aggregates).
   • Set -streamAggr.keepInput=true to also write the matched raw samples. This lets you compare raw vs aggregated queries side-by-side.

2. Stage 2 — verify (see Phase 5). Aggregated output should match the corresponding sum(rate(...)) / histogram_quantile(...) over raw for at least an hour, including across a vmagent restart if possible.

3. Stage 3 — cut over. Remove -streamAggr.keepInput. That's it — the cardinality reduction is realized at this point. The default SA behavior writes aggregates and drops matched raw samples from the remote-write output (only the unmatched-by-rule samples continue to flow as raw). You do not need to drop the raw metric at the scrape side; in fact you must not, because vmagent's pipeline is scrape → relabel → dedup → stream aggregation → remote write, so a scrape-side drop removes the input the aggregator needs and the rule produces nothing.

   Optionally set -streamAggr.dropInput=true only if you also want to drop all unmatched-by-rule samples (rare; appropriate when this remote-write destination is dedicated to aggregates and another remote-write carries the full-resolution data). See Phase 3f for the three input-retention modes.

4. Stage 4 — migrate consumers. Update dashboards / alerts to the new metric name (with the auto-added suffix). The suffix format is :<interval>[_by_<labels>][_without_<labels>]_<output> where both label lists are sorted alphabetically — without: [pod, instance, path] yields _without_instance_path_pod_; by: [function, region, cold_start] yields _by_cold_start_function_region_. For outputs: [quantiles(0.5, 0.99)] each output series also carries a quantile="<phi>" label. If you'd rather keep the original metric name, set keep_metric_names: true (valid only with a single output), and make sure raw is fully dropped or you'll have two series with the same name and different semantics.

Reload without restart:

kill -SIGHUP $(pidof vmagent)
# or
curl -X POST http://vmagent:8429/-/reload

4d. Estimate the savings

For each rule, give the user two numbers:

• Series before (current): from seriesCountByMetricName in TSDB status, or user-reported.
• Series after (measured): the count returned by the count(sum(rate(<metric>[5m])) by (<kept_labels>)) query in Phase 2a, run against current data. This is the authoritative number — the rule will produce roughly this many output series on the current data.

Only fall back to the product-of-labels formula when live data isn't reachable, and label it as an upper bound:

output_series ≤ Π(unique values of each label in `by`, including __name__)

The product overestimates when label values are correlated (most routes don't return every status code) and can also underestimate when there are labels the user didn't enumerate — both reasons to prefer the measured count.

Sanity check: if the measured after-aggregation count is larger than before, the by/without is wrong — you're keeping labels that produce more unique combinations than expected.

Phase 5: Verify

After the rule is live, run these checks. Use them in stage 2 (with raw retained) and again in stage 3.

5a. Stream aggregation internal metrics

Check metrics exposed by vmagent that does the aggregation (or check them on the vmagent Grafana dashboard in Stream aggregation section):

5b. Cross-check aggregated vs raw (stage 2 only)

For a counter aggregated with rate_sum:

# raw
sum by (route, method, le) (rate(http_request_duration_seconds_bucket[5m]))

# aggregated (rate_sum is already a per-second rate — sum it, don't rate it again)
sum by (route, method, le) (http_request_duration_seconds_bucket:30s_without_pod_instance_rate_sum)

The two should track within a few percent. If they diverge sharply, suspect:

• Multi-vmagent collision (Phase 3a, case E).
• Behind a load balancer (Phase 3a, case D).
• Mismatched dedup_interval vs storage -dedup.minScrapeInterval.
• For histograms: sample delays — enable enable_windows and/or raise interval.

For histograms via rate_sum:

# raw p99
histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

# aggregated p99
histogram_quantile(0.99, sum by (le) (avg_over_time(http_request_duration_seconds_bucket:30s_without_pod_instance_rate_sum[5m])))

5c. Sanity checks

• Output series count from count({__name__="<aggregated_name>"}) ≈ your Phase 4d estimate.
• No alerts about histogram_quantile(0.99, sum(rate(vm_streamaggr_flush_duration_seconds_bucket[1m])) by (vmrange)) approaching interval.
• For total/increase outputs: counter is monotonic; resets only on vmagent restart (which is expected unless flush_on_shutdown: true is set).

High Availability

Running multiple vmagent replicas for resilience is supported, but only under specific conditions — stream aggregation maintains state in the aggregator process, and that state cannot be replicated across replicas.

The mechanism

Multiple replicas produce the same output series — identical label sets — and let vmstorage's deduplication collapse the duplicates. This works only when:

1. Every replica receives identical input — same scrape targets, or fan-out (not round-robin) remote-write. A round-robin load balancer in front of HA replicas splits the stream, which both defeats HA and produces incomplete aggregates per replica.
2. Every replica runs identical aggregation configuration.
3. Every replica produces output samples with identical labels (no per-replica tagging — see below).
4. vmstorage has -dedup.minScrapeInterval set (usually equal to the aggregation interval); the per-rule dedup_interval should match.
5. The chosen aggregation output is dedup-safe (next section).

Do not tag HA replicas with distinct labels (-remoteWrite.label=vmagent=A, =B). Distinct labels make the series genuinely different — vmstorage won't dedup, you pay double the storage, and the consumer must sum without (vmagent) everywhere. The "tag each replica" pattern is for sharded topologies (case F), where each replica sees a disjoint slice of inputs and must keep them apart, not for HA where replicas see the same inputs.

HA-safe vs unsafe outputs

For consumers that insist on a counter shape (e.g., legacy alerts using rate(...[5m])), the HA-safe options are:

• Switch the aggregation to rate_sum and update the query (rate_sum is already a per-second rate — sum/avg it instead of taking rate()).
• Switch to increase and use increase-style queries.
• Run a single aggregator (active/passive failover) instead of true HA — accept brief gaps during failover rather than divergent monotonic state.

HA vs sharding — different patterns, don't combine on the same dimension

• Sharded (case F): each replica sees a disjoint slice of inputs. Each emits distinct output series — either by keeping the shard key in by: (preferred — roll up at query time), or by tagging with a unique label. Different replicas produce different values intentionally.
• HA (case E): every replica sees the same inputs and emits the same output labels. Different replicas produce equal values that

…(truncated)