K6 Trend Analysis

k6 Trend Analysis

Analyze metric trends across multiple runs of a Grafana Cloud k6 test to catch degradation early -- before thresholds breach and alerts fire. A P95 at 380ms against a 500ms threshold is "green" today, but if it was 250ms a month ago, something is quietly degrading; this skill surfaces that drift and recommends action.

What this skill does NOT do

• Deep-dive into a single run's failure: load k6-cloud-investigate-test
• Edit scripts or apply threshold changes: load k6-test-maintenance
• Create new test scripts: use the appropriate test creation workflow
• Query service-side metrics directly: this skill hands off to debug-with-grafana when observability correlation is needed

Dependencies

This skill delegates all GCk6 API mechanics to k6-manage. Read it before executing any API call -- it covers auth, path construction (the doubled cloud/cloud/ prefix), pagination with @nextLink, the spill envelope, and metric query syntax. Do not duplicate that knowledge here.

Tools used: gcx (via k6-manage patterns).

Workflow

Follow these steps in order. Present findings at the end -- do not apply changes.

Step 1: Identify the test

The user provides one of:

• A GCk6 URL (extract the load test ID from the path)
• A test ID directly
• A test name (search via gcx k6 tests list or the v6 API)

Confirm the test exists by fetching its metadata. Record the id, name, project_id, and created timestamp. You will need the load test ID (not a run ID) for the multi-run metric endpoints.

Step 2: Determine the analysis window

Default to last 30 days. Adjust if:

• The user requests a specific window
• The test has very few runs (<5 in 30 days) -- widen the window and note this
• The test runs very frequently (hundreds of runs in 30 days) -- consider sampling or narrowing. Ask the user if the volume is extreme (>200 runs)

State the window explicitly: "Analyzing runs from {start_date} to {end_date}."

Step 3: Fetch runs in the window

List all runs for the test using the v6 API with $orderby=created desc pagination (see k6-manage Section 3 for the @nextLink loop pattern). Filter to runs within the analysis window.

For each run, record:

• id, created, ended
• result (passed/failed/timed_out)
• status (created/queued/initializing/running/finished/aborted)
• note (if present -- users sometimes annotate runs)

Discard runs that did not reach finished status (aborted, timed_out, etc.) unless the user specifically asks about them -- incomplete runs produce unreliable metric aggregates.

Count the runs. If fewer than 3 usable runs exist, inform the user that trend analysis is not meaningful with this sample size and suggest widening the window or waiting for more runs.

Step 4: Fetch all metrics and their types

Before querying values, discover what metrics the test emits. Use the multi-run metric listing endpoint (k6-manage references/metrics.md Section 2):

GET /cloud/v5/load_tests/{loadTestId}/metrics(test_run_ids=[{id1},{id2},...])

Pass a representative subset of run IDs (the first and last few) to catch metrics that may have been added or removed over the window. Record each metric's name and type (counter, gauge, trend, rate).

Group metrics by type -- the query method must match the metric type (see metrics.md "Query methods"). Using the wrong method returns empty results.

Step 5: Fetch per-run aggregate values

For each metric, query its aggregate value across all runs in the window using the multi-run aggregate endpoint (metrics.md Section 8):

GET /cloud/v5/load_tests/{loadTestId}/query_aggregate_k6(
  query='<method>',
  metric='<metric_name>',
  test_run_ids=[{id1},{id2},...]
)

Concrete gcx form (proxy prefix per k6-manage §2; -o json avoids the spill envelope):

LT=<load_test_id>; IDS="123,124,125"
gcx --context <ctx> api "/api/plugins/k6-app/resources/cloud/cloud/v5/load_tests/$LT/query_aggregate_k6(query='histogram_quantile(0.95)',metric='http_req_duration',test_run_ids=[$IDS])" -o json

Choose the aggregate method based on metric type:

For trend-type metrics (latencies), query multiple quantiles (P50, P90, P95, P99) to see if degradation is uniform or concentrated in the tail.

Always break down by request grouping for multi-target tests. A single aggregate p95 across a whole test obscures regressions confined to one endpoint or one page -- a 3x slowdown on one URL can be invisible at the test-level p95 if the test hits many URLs. Default to grouped queries:

Discover available labels first via the labels endpoint (metrics.md §4) and label-values endpoint (metrics.md §5) before constructing the grouped query. Don't assume labels from reading the script -- a tag rename or refactor can silently shift the label space. Example:

# List labels for a specific metric on a representative run
GET /cloud/v5/test_runs/{id}/labels?match[]=browser_web_vital_lcp

# Then enumerate values for a useful label
GET /cloud/v5/test_runs/{id}/label/url/values

Only fall back to the bare aggregate (no by) for metrics where grouping adds no information -- e.g., vus, load_generator_cpu_percent, single-target tests where every request hits the same URL.

The response includes test_run_id as a label -- use this to map each value back to its run timestamp from Step 3. Grouped queries return one series per (group, run) combination; flatten into a tidy "rid x group" table for the trend computation in Step 7.

If there are too many run IDs to fit in a single URL (hundreds), batch the queries into groups of 50 run IDs and merge the results.

Step 6: Extract thresholds

Thresholds define what "passing" means. Fetch the test's current script (via k6-manage Section 5) and parse the export const options = { thresholds: {...} } block. Record each threshold's:

• Metric name and selector (e.g., http_req_duration{name:homepage})
• Condition (e.g., p(95)<500)
• Whether abortOnFail is set

Also check the most recent run's threshold results from the run data to see which thresholds are currently passing vs. failing.

Not all metrics will have thresholds -- that's fine. Metrics without thresholds still get trend analysis; they just won't have headroom calculations.

Step 7: Compute trends

Detect inflection points and rule out script changes deterministically. Before drawing conclusions about a regression, look for discontinuities in the metric values -- sudden jumps or drops that align across multiple metrics on the same date. When you spot one, do not guess whether the script changed. The run-bundled script endpoint gives a deterministic answer in seconds:

1. sha256-diff the bundled scripts at the boundary. Fetch the snapshot from the last "before" run and the first "after" run via GET /cloud/v6/test_runs/{id}/script (k6-manage §5, "Two distinct script endpoints" -- use the run-scoped endpoint, not the load-test one, since the latter only shows the current version). Compare with shasum -a 256.

   gcx --context <ctx> api /api/plugins/k6-app/resources/cloud/cloud/v6/test_runs/<before_id>/script > /tmp/before.bin
   gcx --context <ctx> api /api/plugins/k6-app/resources/cloud/cloud/v6/test_runs/<after_id>/script > /tmp/after.bin
   shasum -a 256 /tmp/before.bin /tmp/after.bin
   

   For thoroughness on multi-run inflections, hash every run across the transition -- if N consecutive runs share one hash and N more share another, you have a clean before/after boundary. If hashes change mid-stream, the test was edited multiple times.

2. If the sha256 differs, the test script changed -- the inflection may be a test-side artifact, not a service regression. Split the analysis into distinct eras at the boundary and compute trends within each era separately. Comparing metrics across script changes produces misleading trends -- a P95 drop from 3,000ms to 150ms is not an "improvement" if the script simply stopped hitting a slow endpoint. State the eras explicitly in the report and focus recommendations on the most recent era.

3. If the sha256 matches, the script is byte-identical and the regression is external to the test (service-side, infrastructure, or load-zone). This is high-confidence information -- carry it into Step 9 (service-side correlation) instead of leaving "did the test change?" as an open question.

This sha256 diff is a 5-second deterministic check that rules out a huge class of causes. Run it at every detected inflection, not just when the user asks.

For each metric, build a time-ordered series of (run_timestamp, value) pairs. Then compute:

Basic statistics:

• Mean, standard deviation, min, max across all runs
• Current value (most recent run)
• Baseline value (oldest run in window, or mean of first 3 runs for stability)

Trend direction: Split the runs into two halves (first half and second half of the time window). Compare the mean of each half:

• Degrading: second-half mean is worse by >10% (higher for latency/errors, lower for success rates)
• Improving: second-half mean is better by >10%
• Stable: change is within 10%
• Volatile: standard deviation exceeds 25% of the mean, regardless of direction

The 10% and 25% thresholds are starting points. If the user's test has very tight tolerances or very noisy metrics, adjust and explain the reasoning.

Rate of change: Express as percentage change per week: ((recent_mean - baseline_mean) / baseline_mean) * 100 / weeks_in_window. This normalizes across different window sizes.

Anomaly detection: Flag any run where the metric value is more than 2 standard deviations from the overall mean. These are potential inflection points worth investigating individually.

Step 8: Headroom analysis

For metrics that have thresholds defined (from Step 6), compute headroom:

headroom_pct = ((threshold_value - current_value) / threshold_value) * 100

Classify headroom:

• Comfortable (>50%): well within limits
• Adequate (20-50%): healthy but worth monitoring
• Thin (<20%): at risk of breaching if trends continue
• Breached (<0%): already failing

For degrading metrics with thin headroom, estimate when the threshold will be breached if the current rate of change continues:

weeks_until_breach = headroom_absolute / rate_of_change_per_week

This is a rough projection, not a prediction -- present it as "at the current rate of degradation, this metric could breach in approximately N weeks."

Step 9: Service-side correlation (when warranted)

If Step 7 reveals degradation, offer (don't auto-run) to correlate with service-side data to separate service degradation (fix the service), test-environment changes (load-zone latency, LG exhaustion), and script changes (a slower edit). Present findings and ask first.

Hand off to debug-with-grafana (via gcx) to query the service's Prometheus metrics and Loki logs for the same window; look for service error-rate changes, upstream latency, resource pressure (CPU/memory/pools), and deploys coinciding with inflection points.

When the service has no observability (third-party or another team's service -- no Prometheus job, Loki stream, or probe), fall back to client-side signal, which still localises the regression:

• Browser tests: compare Tempo iteration traces (k6-manage §7, /api/v1/tempo/api/search + /traces/{id}) for a before/after run. The span-name rollup, slowest spans, and per-URL navigation durations pin the regression to a URL, locator action, or asset; web vitals are web_vital.* span attributes -- read them directly.
• Protocol tests: compare http_req_duration by (name, status) and http_req_failed by (name, status); a regression on one named request points at that endpoint. Split http_req_waiting (server time) vs http_req_receiving (transfer) to separate slow processing from slow download.
• Payload corroboration (both): query data_received / browser_data_received. Latency up + payload up -> server content changes; latency up + payload stable -> server processing changes; intermittent payload -> flaky cache or A/B test.

Prefer server-side correlation when available -- it answers "why" directly; the fallbacks answer "where" and "what kind of change", and make a good ticket for the owning team.

Step 10: Present the report

Always present findings as a structured report. Never apply changes directly.

Report template

Use this structure. Omit sections that don't apply (e.g., skip "Anomalies" if none were detected).

# Trend Analysis: {test_name}

**Test ID**: {test_id}
**Analysis window**: {start_date} to {end_date} ({N} runs analyzed)
**Overall health**: {Healthy | Watch | Degrading | Critical}

## Run Summary

| Period | Runs | Passed | Failed | Pass Rate |
|--------|------|--------|--------|-----------|
| First half | N | N | N | N% |
| Second half | N | N | N | N% |

## Metric Trends

| Metric | Type | Current | Baseline | Change | Trend | Threshold | Headroom |
|--------|------|---------|----------|--------|-------|-----------|----------|
| http_req_duration (P95) | trend | 380ms | 250ms | +52% | Degrading | 500ms | 24% |
| http_req_failed | rate | 0.8% | 0.3% | +167% | Degrading | 1% | 20% |
| http_reqs | counter | 15,230 | 15,100 | +0.9% | Stable | - | - |

## Flagged Issues

### 1. {metric_name}: {classification}
- **Current**: {value} | **Baseline**: {value} | **Change**: {pct}%
- **Threshold**: {threshold} | **Headroom**: {pct}%
- **Rate of change**: {pct}% per week
- **Projected breach**: ~{N} weeks at current rate
- **Anomalous runs**: {run_ids with dates, if any}

## Threshold Recommendations

| Metric | Current Threshold | Recommended | Rationale |
|--------|-------------------|-------------|-----------|
| http_req_duration | p(95)<500 | p(95)<420 | Current P95 is 380ms; tightening to 420ms gives 10% headroom from current performance while surfacing further degradation early |

## Suggested Next Steps

- [ ] **Investigate service side**: P95 latency has increased 52% -- consider
      loading `debug-with-grafana` to check service health
- [ ] **Deep-dive run {run_id}**: anomalous P95 spike on {date} -- consider
      loading `k6-cloud-investigate-test` for this run
- [ ] **Tighten thresholds**: 2 metrics have >30% headroom that could be
      tightened -- consider loading `k6-test-maintenance` to apply changes

Overall health classification

Derive the overall health from the worst-case metric:

• Healthy: all metrics stable or improving, headroom comfortable or adequate
• Watch: at least one metric degrading but headroom still adequate
• Degrading: at least one metric degrading with thin headroom
• Critical: at least one metric has breached its threshold, or multiple metrics are degrading with thin headroom

Threshold recommendations

When recommending threshold changes, follow these principles:

• Only tighten, never loosen unless the user asks. Loosening thresholds masks problems.
• Target 10-20% headroom above the recent P95 of the metric. Enough room for normal variance but tight enough to catch real degradation.
• Use the second-half mean as the baseline, not the single most recent run (which could be an outlier).
• Respect the user's intent: if thresholds are currently very loose (>100% headroom), they may be intentionally permissive. Mention the opportunity to tighten but don't push hard -- the user knows their context.
• Consider grouped thresholds: if the test uses tag-based thresholds (e.g., http_req_duration{name:homepage}), recommend per-endpoint thresholds where the trends differ between endpoints.

Gotchas