Prometheus Knowledge Patch

Prometheus Knowledge Patch

Use this skill when upgrading, configuring, querying, integrating, or operating Prometheus and recent behavior may affect the result. Inspect the deployed server, image, configuration, and client versions first. Apply version-tagged guidance only when it matches the deployment, and prefer observed behavior and the repository's tests over assumptions.

Reference index

Critical upgrade and security decisions

Patch vulnerable lines before exposing endpoints

• On the 3.11 line, deploy at least 3.11.3. Earlier patch levels can disclose an AzureAD client secret, accept oversized Snappy remote-read requests, and expose stored-XSS paths in the UIs.
• Upgrade STACKIT service-discovery users to a fixed 3.12 release because rendered configuration previously exposed credentials.
• Prometheus 3.13.0 fixes CVE-2026-44990 in the UI and strips credentials and configured headers when an HTTP redirect crosses hosts.
• Use 3.13.2 or later when CVE-2026-56852 or GHSA-hrxh-6v49-42gf applies.

Read Migration, security, and deployment before choosing an image or patch level.

Treat the v3 transition as a compatibility migration

• Remove feature gates that became default and replace the former agent and remote-write-receiver gates with --agent and --web.enable-remote-write-receiver.
• Remove the deleted storage.tsdb.allow-overlapping-blocks, alertmanager.timeout, and storage.tsdb.retention command-line flags.
• Upgrade through v2.55 before reusing a TSDB with v3. A v3 data directory cannot be downgraded below v2.55 without abandoning that persistent data.
• Configure Alertmanager API v2 and use Alertmanager 0.16.0 or later.
• Scrapes with a missing, unparsable, or unknown Content-Type now fail. Fix the producer or configure fallback_scrape_protocol deliberately.
• UTF-8 metric and label names are accepted by default. Set metric_name_validation_scheme: legacy globally or per job when old name validation is required.
• Update log parsing for log/slog fields such as time, source, and uppercase level, and update selectors for normalized le and quantile values such as "1.0".

See Migration, security, and deployment for the complete transition checklist.

Histograms, storage, and downgrade boundaries

Native histogram activation is configuration-driven

Native histograms are stable starting in v3.9, so native-histograms becomes a no-op, but scraping them still requires:

global:
  scrape_native_histograms: true

Use always_scrape_classic_histograms to retain concurrently exposed classic histograms; the old scrape_classic_histograms key is obsolete. Global, per-job, and per-target controls are available, with target relabel labels providing the narrowest override.

Start-timestamp storage changes data compatibility

st-storage requires XOR2 for float chunks. Native- and float-histogram start timestamps additionally require histograms-st-encoding. The resulting WAL records need Prometheus 3.11 or later, and XOR2 or histogram-ST blocks may be unreadable by older servers and downstream block consumers.

st-synthesis rewrites scraped cumulative values: it drops the first sample and subtracts it from later samples. It does not apply to remote write or OTLP, and it changes out-of-order handling and recovery after append failures.

Before enabling these features, read Histograms, TSDB, and start timestamps.

PromQL compatibility quick reference

• Regular-expression . matches newlines as well as other characters.
• Duration and float literals are interchangeable, and duration expressions later become default. Use current min_of() and max_of() names instead of the earlier experimental duration min() and max() forms.
• Millisecond range selectors retain their precision rather than rounding to whole seconds.
• histogram_fraction() and histogram_quantile() return no value when classic and native histograms coexist at the same timestamp.
• fill(), fill_left(), and fill_right() supply values for unmatched sides of a binary expression; later fixes restore group-matched range results.
• Start-timestamp-aware rates cannot be combined with anchored or smoothed extended selectors.
• sort(), sort_by_label(), and sort_by_label_desc() do nothing in range queries and now emit warnings.
• Query raw OTLP deltas with aligned sum_over_time() windows; counter functions are wrong for natively stored delta values.

Read PromQL before changing rule expressions or relying on experimental syntax.

Remote storage and OTLP quick reference

• Remote-write HTTP/2 now defaults off. Set http_config.enable_http2: true only when retaining the previous behavior is intentional.
• Replace deprecated remote-write input and highest-timestamp metrics with the WAL-watcher, dropped-sample, and queue-highest-timestamp metrics.
• Remote Write 2 uses “start timestamp” terminology and carries supported start timestamps, type labels, and unit labels.
• Too-old Remote Write 2 histogram samples return HTTP 400 so senders do not retry them indefinitely.
• Choose one OTLP delta mode: stateful delta-to-cumulative conversion or raw native delta ingestion. They are mutually exclusive.
• Validate translated label collisions through prometheus_api_otlp_translation_warnings_total.

Use Remote storage and OTLP for translation strategies, identity propagation, authentication, and receiver limits.

Scraping, configuration, and rules quick reference

• Explicitly configure scrape_protocols when created-timestamp zero injection must not make protobuf the first negotiation choice.
• Use extra_scrape_metrics: true globally or per job; the old extra-scrape-metrics feature gate is deprecated.
• Configuration reloads honor histogram scrape settings and automatic reload watches referenced rule and scrape files. Automatic reload is stable.
• Invalid remote-write queue settings are rejected while loading configuration.
• Unknown rule dependencies evaluate serially; optional concurrent evaluation applies only to dependency-free rules and should be bounded with --rules.max-concurrent-evals.
• API and UI clients must handle the explicit unknown state for an alerting rule that has not yet evaluated.

See Scraping, configuration, and rules for relabel overrides, alert delivery, reload cleanup, and template helpers.

APIs and operational monitoring

• Discover server capabilities at /api/v1/features and the machine-readable HTTP contract at /api/v1/openapi.yaml.
• Query endpoints accept limit. For stats, use only true or all; other values are deprecated and scheduled for rejection.
• Distinguish samplesRead storage I/O from totalQueryableSamples evaluator loads. Enable promql-per-step-stats with stats=all for per-step reads.
• The TSDB status endpoint caps statistics at 10,000 sets.
• Monitor WAL/WBL unknown references, out-of-order distance, stale series, native histogram head use, OTLP translation warnings, and service-discovery update timestamps with their dedicated metrics.
• Notification metrics gain an alertmanager dimension, and each Alertmanager now has an independent send loop.

Read HTTP APIs and observability before updating clients, dashboards, alerts, or tracing integrations.

Service discovery and tooling

• Audit relabeling whenever a provider changes metadata labels. In particular, Hetzner hcloud no longer exposes __meta_hetzner_datacenter.
• Use provider-side filters where supported to reduce target volume before relabeling.
• promtool debug output is on stderr. Keep stdout separate in pipelines.
• Match promtool feature flags to gated PromQL syntax during checks, and use its fuzzy comparisons and explicit test start timestamps for time-sensitive rules.
• Prefer the feature discovery API over guessing support from a version string.

Consult Service discovery and Promtool, UI, and feature flags for provider-specific and command-specific details.

Working method

1. Identify the exact server version, image variant, enabled features, storage formats, remote protocol versions, and relevant client versions.
2. Read only the topic references needed for the task and honor their inline version attributions.
3. For upgrades, check security floors, removed options, storage downgrade boundaries, protocol changes, metric renames, and label-shape changes.
4. Validate configurations and rules with the matching promtool feature gates, then exercise affected API clients and dashboards.
5. Treat experimental formats and syntax as changeable. Preserve a rollback path before enabling storage encodings or ingestion modes that rewrite data.