# Prometheus Knowledge Patch > Prometheus - Skill: `nevaberry/prometheus-knowledge-patch-2` (Agent Skill, multi-file: 10 files) - Install (CLI): `npx skillmds@latest add nevaberry/prometheus-knowledge-patch-2` - Raw SKILL.md: https://api.skillmd.com/api/skills/nevaberry/prometheus-knowledge-patch-2/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - License: MIT - Author: nevaberry (https://skillmd.com/u/nevaberry) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/nevaberry/prometheus-knowledge-patch-2 --- # Prometheus Knowledge Patch Use this skill when upgrading, configuring, querying, integrating, or operating Prometheus and recent behavior may affect the result. Inspect the deployed server, image, configuration, and client versions first. Apply version-tagged guidance only when it matches the deployment, and prefer observed behavior and the repository's tests over assumptions. ## Reference index | Reference | Topics | |---|---| | [Histograms, TSDB, and start timestamps](references/histograms-tsdb-start-timestamps.md) | Native and classic histograms, TSDB compatibility, retention, WAL, encodings, start timestamps | | [HTTP APIs and observability](references/http-apis-observability.md) | Query and status APIs, OpenAPI, self-metrics, tracing, notification and service metrics | | [Migration, security, and deployment](references/migration-security-deployment.md) | Major-upgrade breaks, security floors, containers, logs, removed flags, redirect credentials | | [PromQL](references/promql.md) | Language changes, experimental syntax, functions, modifiers, annotations, result corrections | | [Promtool, UI, and feature flags](references/promtool-ui-feature-flags.md) | Validation and test tooling, UI behavior, feature discovery, current feature-gate constraints | | [Remote storage and OTLP](references/remote-storage-otlp.md) | Remote read/write, protocol versions, authentication, OTLP translation and delta ingestion | | [Scraping, configuration, and rules](references/scraping-configuration-rules.md) | Scrape negotiation, relabeling, reloads, rule evaluation, alerts, templates | | [Service discovery](references/service-discovery.md) | Provider additions, metadata labels, filters, identity, reliability, build tags | ## Critical upgrade and security decisions ### Patch vulnerable lines before exposing endpoints - On the 3.11 line, deploy at least 3.11.3. Earlier patch levels can disclose an AzureAD client secret, accept oversized Snappy remote-read requests, and expose stored-XSS paths in the UIs. - Upgrade STACKIT service-discovery users to a fixed 3.12 release because rendered configuration previously exposed credentials. - Prometheus 3.13.0 fixes CVE-2026-44990 in the UI and strips credentials and configured headers when an HTTP redirect crosses hosts. - Use 3.13.2 or later when CVE-2026-56852 or GHSA-hrxh-6v49-42gf applies. Read [Migration, security, and deployment](references/migration-security-deployment.md) before choosing an image or patch level. ### Treat the v3 transition as a compatibility migration - Remove feature gates that became default and replace the former `agent` and `remote-write-receiver` gates with `--agent` and `--web.enable-remote-write-receiver`. - Remove the deleted `storage.tsdb.allow-overlapping-blocks`, `alertmanager.timeout`, and `storage.tsdb.retention` command-line flags. - Upgrade through v2.55 before reusing a TSDB with v3. A v3 data directory cannot be downgraded below v2.55 without abandoning that persistent data. - Configure Alertmanager API v2 and use Alertmanager 0.16.0 or later. - Scrapes with a missing, unparsable, or unknown `Content-Type` now fail. Fix the producer or configure `fallback_scrape_protocol` deliberately. - UTF-8 metric and label names are accepted by default. Set `metric_name_validation_scheme: legacy` globally or per job when old name validation is required. - Update log parsing for `log/slog` fields such as `time`, `source`, and uppercase `level`, and update selectors for normalized `le` and `quantile` values such as `"1.0"`. See [Migration, security, and deployment](references/migration-security-deployment.md) for the complete transition checklist. ## Histograms, storage, and downgrade boundaries ### Native histogram activation is configuration-driven Native histograms are stable starting in v3.9, so `native-histograms` becomes a no-op, but scraping them still requires: ```yaml global: scrape_native_histograms: true ``` Use `always_scrape_classic_histograms` to retain concurrently exposed classic histograms; the old `scrape_classic_histograms` key is obsolete. Global, per-job, and per-target controls are available, with target relabel labels providing the narrowest override. ### Start-timestamp storage changes data compatibility `st-storage` requires XOR2 for float chunks. Native- and float-histogram start timestamps additionally require `histograms-st-encoding`. The resulting WAL records need Prometheus 3.11 or later, and XOR2 or histogram-ST blocks may be unreadable by older servers and downstream block consumers. `st-synthesis` rewrites scraped cumulative values: it drops the first sample and subtracts it from later samples. It does not apply to remote write or OTLP, and it changes out-of-order handling and recovery after append failures. Before enabling these features, read [Histograms, TSDB, and start timestamps](references/histograms-tsdb-start-timestamps.md). ## PromQL compatibility quick reference - Regular-expression `.` matches newlines as well as other characters. - Duration and float literals are interchangeable, and duration expressions later become default. Use current `min_of()` and `max_of()` names instead of the earlier experimental duration `min()` and `max()` forms. - Millisecond range selectors retain their precision rather than rounding to whole seconds. - `histogram_fraction()` and `histogram_quantile()` return no value when classic and native histograms coexist at the same timestamp. - `fill()`, `fill_left()`, and `fill_right()` supply values for unmatched sides of a binary expression; later fixes restore group-matched range results. - Start-timestamp-aware rates cannot be combined with `anchored` or `smoothed` extended selectors. - `sort()`, `sort_by_label()`, and `sort_by_label_desc()` do nothing in range queries and now emit warnings. - Query raw OTLP deltas with aligned `sum_over_time()` windows; counter functions are wrong for natively stored delta values. Read [PromQL](references/promql.md) before changing rule expressions or relying on experimental syntax. ## Remote storage and OTLP quick reference - Remote-write HTTP/2 now defaults off. Set `http_config.enable_http2: true` only when retaining the previous behavior is intentional. - Replace deprecated remote-write input and highest-timestamp metrics with the WAL-watcher, dropped-sample, and queue-highest-timestamp metrics. - Remote Write 2 uses “start timestamp” terminology and carries supported start timestamps, type labels, and unit labels. - Too-old Remote Write 2 histogram samples return HTTP 400 so senders do not retry them indefinitely. - Choose one OTLP delta mode: stateful delta-to-cumulative conversion or raw native delta ingestion. They are mutually exclusive. - Validate translated label collisions through `prometheus_api_otlp_translation_warnings_total`. Use [Remote storage and OTLP](references/remote-storage-otlp.md) for translation strategies, identity propagation, authentication, and receiver limits. ## Scraping, configuration, and rules quick reference - Explicitly configure `scrape_protocols` when created-timestamp zero injection must not make protobuf the first negotiation choice. - Use `extra_scrape_metrics: true` globally or per job; the old `extra-scrape-metrics` feature gate is deprecated. - Configuration reloads honor histogram scrape settings and automatic reload watches referenced rule and scrape files. Automatic reload is stable. - Invalid remote-write queue settings are rejected while loading configuration. - Unknown rule dependencies evaluate serially; optional concurrent evaluation applies only to dependency-free rules and should be bounded with `--rules.max-concurrent-evals`. - API and UI clients must handle the explicit `unknown` state for an alerting rule that has not yet evaluated. See [Scraping, configuration, and rules](references/scraping-configuration-rules.md) for relabel overrides, alert delivery, reload cleanup, and template helpers. ## APIs and operational monitoring - Discover server capabilities at `/api/v1/features` and the machine-readable HTTP contract at `/api/v1/openapi.yaml`. - Query endpoints accept `limit`. For `stats`, use only `true` or `all`; other values are deprecated and scheduled for rejection. - Distinguish `samplesRead` storage I/O from `totalQueryableSamples` evaluator loads. Enable `promql-per-step-stats` with `stats=all` for per-step reads. - The TSDB status endpoint caps statistics at 10,000 sets. - Monitor WAL/WBL unknown references, out-of-order distance, stale series, native histogram head use, OTLP translation warnings, and service-discovery update timestamps with their dedicated metrics. - Notification metrics gain an `alertmanager` dimension, and each Alertmanager now has an independent send loop. Read [HTTP APIs and observability](references/http-apis-observability.md) before updating clients, dashboards, alerts, or tracing integrations. ## Service discovery and tooling - Audit relabeling whenever a provider changes metadata labels. In particular, Hetzner `hcloud` no longer exposes `__meta_hetzner_datacenter`. - Use provider-side filters where supported to reduce target volume before relabeling. - `promtool` debug output is on stderr. Keep stdout separate in pipelines. - Match `promtool` feature flags to gated PromQL syntax during checks, and use its fuzzy comparisons and explicit test start timestamps for time-sensitive rules. - Prefer the feature discovery API over guessing support from a version string. Consult [Service discovery](references/service-discovery.md) and [Promtool, UI, and feature flags](references/promtool-ui-feature-flags.md) for provider-specific and command-specific details. ## Working method 1. Identify the exact server version, image variant, enabled features, storage formats, remote protocol versions, and relevant client versions. 2. Read only the topic references needed for the task and honor their inline version attributions. 3. For upgrades, check security floors, removed options, storage downgrade boundaries, protocol changes, metric renames, and label-shape changes. 4. Validate configurations and rules with the matching `promtool` feature gates, then exercise affected API clients and dashboards. 5. Treat experimental formats and syntax as changeable. Preserve a rollback path before enabling storage encodings or ingestion modes that rewrite data.