Prometheus Knowledge Patch

Prometheus Knowledge Patch

Use this skill when designing, upgrading, configuring, querying, or operating Prometheus. Establish the deployed Prometheus version and enabled feature flags before applying version-sensitive behavior. Prefer the running server's feature API, configuration, and observed behavior when they disagree with assumptions.

Reference index

Upgrade and compatibility triage

Before crossing the major-version boundary

• Remove feature flags whose behavior became default and replace the old agent and remote-write-receiver flags with their dedicated command-line switches.
• Remove deleted TSDB retention, overlapping-block, and Alertmanager timeout flags before starting the new binary.
• Upgrade through the compatible TSDB transition release before reusing a data directory. Treat newer block and WAL encodings as downgrade boundaries.
• Upgrade Alertmanager and use its v2 API.
• Update scrape endpoints or configure an explicit fallback protocol; missing or unknown Content-Type values are fatal.
• Audit regular expressions, UTF-8 name handling, normalized le and quantile labels, and structured-log parsers.
• Read Migration, security, and operations and Scraping and ingestion before changing production startup or scrape configuration.

Security floor

• Do not remain on an affected patch level when the references identify a credential disclosure, stored XSS, request-size validation, redirect-header, or dependency advisory.
• Treat /-/config, UI exposure, remote-read inputs, cross-host redirects, and service-discovery credentials as security-sensitive surfaces.
• For the 3.11 line, deploy the patched maintenance release identified in the operations reference; later lines have additional security fixes.

Regression-sensitive releases

• Avoid a release with the Agent startup crash and broken scrape relabel keep/drop behavior; use its maintenance fix.
• Upgrade deployments using XOR2, early stale-series compaction, out-of-order ingestion, or native histograms when the storage reference identifies restart, replay, compaction, or sample-loss fixes.

Feature-flag decisions

Verify support rather than inferring it

Use /api/v1/features when available. A flag may be promoted to stable and become a no-op, renamed, or replaced by configuration. Pass matching feature flags to promtool when validating gated syntax.

Current high-impact transitions

• Native histogram support is stable, but scraping native histograms still requires scrape configuration. Retaining classic histograms and converting classic buckets are separate controls available globally, per job, and per target.
• Duration expressions and first_over_time have become stable. Use the current experimental-function flag spelling for functions that remain unstable.
• Automatic configuration reload is stable; it also watches referenced rule and scrape configuration files.
• Extra scrape metrics moved from a feature flag to scrape configuration.
• Out-of-order native histograms follow the out-of-order window and histogram settings; their former dedicated flag is a no-op.

Experimental storage features

Do not enable start-timestamp storage, start-timestamp synthesis, XOR2, histogram start-timestamp encodings, direct I/O, fast startup, or stale-series compaction without reading Histograms, TSDB, and storage. Some combinations change stored values or formats, constrain downgrade paths, or require paired encodings.

Experimental query features

Before using extended range selectors, query-boundary functions, fill modifiers, metadata labels, or concurrent rule evaluation, read PromQL, rules, and templates. Function allowlists, subquery restrictions, rule offsets, reserved labels, and concurrency limits materially affect correctness.

Scraping and histogram quick reference

Native and classic histogram controls

global:
  scrape_native_histograms: true
  always_scrape_classic_histograms: true
  convert_classic_histograms_to_nhcb: true

Use only the controls required by the deployment. Per-target relabel labels can override all three behaviors. Mixed classic and native samples at one timestamp can make some histogram functions return no value.

Protocol and naming compatibility

• Fix exporters to return an accepted protobuf, Prometheus text, or OpenMetrics content type; use fallback_scrape_protocol only deliberately.
• Set metric_name_validation_scheme: legacy globally or per job when an upgrade must preserve legacy name rejection.
• Configure scrape_protocols explicitly when created-timestamp zero injection must not change the default protocol preference.
• Use the requested escaping scheme and per-target controls when different exporters need different behavior.

PromQL and rules quick reference

• A regex dot matches newlines; audit broad matchers after upgrading.
• Duration literals work as scalars, duration expressions support arithmetic, and the newest experimental helpers are named min_of() and max_of().
• Histogram-aware functions have function-specific output typing and omission rules. Do not assume float behavior for mixed vectors.
• anchored and smoothed selectors accept only specific rate/reset functions, reject subqueries, and require rule offsets for reliable smoothed evaluation.
• Raw OTLP deltas are not counters. Query them with aligned sum_over_time(...) windows rather than rate() or increase().
• Range-query sorting functions have no effect and now warn.
• Concurrent rule evaluation adds query load independently of the ordinary query concurrency settings.

Remote I/O quick reference

• Remote-write HTTP/2 is opt-in. Enable it explicitly only when retaining that transport behavior is desired.
• Remote-read implementations must enforce requested selectors and return only matching results.
• Remote Write 1.0 cannot transport custom-bucket native histograms; use a compatible protocol or expect them to be rejected with a warning.
• Remote Write 2 uses start-timestamp terminology and has specific rejection semantics for too-old histogram samples.
• Validate queue configuration during configuration loading and migrate from deprecated remote-write self-metrics.
• Cross-host redirects strip credentials and configured authorization headers.

OTLP choice points

Choose one delta strategy:

• otlp-deltatocumulative maintains in-memory cumulative state and resets it across restarts.
• Native delta ingestion stores raw deltas, is mutually exclusive with conversion, and requires delta-aware queries and explicit stream labeling.

Then decide translation strategy, resource and scope metadata promotion, reserved type/unit labels, start-timestamp handling, and explicit-histogram conversion. Read OTLP ingestion before combining these switches.

Validation workflow

1. Determine the binary version, image variant, feature flags, and runtime configuration.
2. Read only the references relevant to the planned change.
3. Run promtool check config and promtool check rules, passing required syntax flags. Account for stdout/stderr behavior in pipelines.
4. Inspect /api/v1/features, status APIs, target relabel traces, query warnings, and the relevant self-metrics.
5. For storage changes, test restart, replay, compaction, and downgrade or downstream-reader compatibility on a copy of the data.
6. For query changes, compare annotations, histogram typing, sample limits, and read statistics against representative production expressions.