Prometheus

Prometheus

Choose the right metric type, name it clearly, label it sparingly. Prometheus is a pull-based monitoring system built on a dimensional data model — every metric is a time series identified by a name and key-value label pairs. Getting this right at instrumentation time prevents rework later.

References

• Metric types — [${CLAUDE_SKILL_DIR}/references/metric-types.md] Extended type comparison, histogram bucket tuning, summary configuration
• Naming — [${CLAUDE_SKILL_DIR}/references/naming.md] Full naming examples, base units table, character rules, label best practices
• Instrumentation — [${CLAUDE_SKILL_DIR}/references/instrumentation.md] Code patterns per system type, library instrumentation, performance tuning
• PromQL — [${CLAUDE_SKILL_DIR}/references/promql.md] Full operator catalog, vector matching, over-time aggregation, operator precedence
• Alerting and rules — [${CLAUDE_SKILL_DIR}/references/alerting-and-rules.md] Alert design, recording rule naming, aggregation patterns, anti-patterns
• Exporters — [${CLAUDE_SKILL_DIR}/references/exporters.md] Exporter architecture, collectors, help strings, push-based sources

Metric Type Selection

Choose correctly at instrumentation time — changing later requires migrating dashboards, alerts, and recording rules.

Counter

Monotonically increasing value — resets to zero only on restart.

• Use for: requests served, errors occurred, bytes transferred, tasks completed
• API: inc(), inc(v) where v >= 0
• Always suffix with _total: http_requests_total
• Always apply rate() or increase() in queries — raw values are meaningless
• Prometheus handles counter resets automatically in rate()
• Never use a counter for values that can decrease — that is a gauge

Gauge

Value that goes up and down arbitrarily.

• Use for: temperature, memory usage, in-progress requests, queue depth, timestamps
• API: inc(), dec(), set(v), set_to_current_time()
• No _total suffix
• Never apply rate() to a gauge — use deriv() or delta()
• Timestamp pattern: store Unix epoch seconds as myapp_last_success_timestamp_seconds; compute elapsed time with time() - metric in PromQL
• Info metric pattern: myapp_build_info{version="1.2.3", commit="abc"} 1 — metadata as labels with constant value 1

Histogram

Samples observations into configurable buckets. Produces _bucket{le="..."}, _sum, _count.

• Use for: request latencies, response sizes, any distribution needing percentiles or cross-instance aggregation
• API: observe(v)
• Buckets are cumulative — le="0.5" includes all observations <= 0.5
• Must include +Inf bucket (equal to _count)
• Choose buckets matching expected value range; place more buckets near SLO boundaries
• Buckets cannot be changed after metric creation
• Use histogram_quantile() in PromQL to calculate percentiles
• Aggregatable across instances — the primary advantage over summary

Summary

Calculates streaming quantiles on the client side. Produces {quantile="..."}, _sum, _count.

• Cannot be aggregated across instances — avg(x{quantile="0.95"}) is statistically invalid
• _sum and _count without quantiles is a valid and useful configuration

Use summary over histogram only when ALL of these are true:

1. You need accurate quantiles (not approximate)
2. From a single instance (no cross-instance aggregation)
3. You know the exact quantiles at instrumentation time
4. You accept that adding new quantiles requires code changes

Default choice: histogram. See ${CLAUDE_SKILL_DIR}/references/metric-types.md for detailed comparison.

Naming

Format: <namespace>_<subsystem>_<name>_<unit>_<suffix>. Not all parts required — minimum is namespace + meaningful name + unit/suffix.

Naming Rules

1. Use snake_case — lowercase with underscores, matching [a-zA-Z_:][a-zA-Z0-9_:]*
2. Colons (:) are reserved for recording rules — never use in direct instrumentation
3. Double underscore prefix (__) is reserved for Prometheus internals
4. Every metric MUST have a namespace prefix identifying its origin
5. Always use base units — seconds not milliseconds, bytes not megabytes. Let visualization tools handle conversion.
6. Append unit to metric name in plural form: http_request_duration_seconds
7. Suffix counters with _total, counter-with-unit as _<unit>_total (e.g., process_cpu_seconds_total)
8. Suffix info metrics with _info, timestamps with _timestamp_seconds
9. A metric MUST represent the same logical thing across all its label dimensions — sum() or avg() across all dimensions should be meaningful. If nonsensical, split into separate metrics.

See ${CLAUDE_SKILL_DIR}/references/naming.md for base units table, component ordering, and full examples.

Labels

Use labels for dimensions you will filter or aggregate by: http_requests_total{method="GET", status="200"} — not separate metrics per status. Do not put label names in metric names.

When NOT to Use Labels

• Unbounded values — user IDs, email addresses, full URLs, query strings
• High cardinality — anything above ~100 unique values per metric

Cardinality

Every unique label combination is a new time series — each costs RAM, CPU, disk, and network. Cardinality math: total series = metric cardinality x number of targets.

• < 10: Safe for most metrics
• 10-100: Acceptable, monitor growth
• 100-1000: Investigate alternatives
• > 1000: Move analysis out of Prometheus

Label Best Practices

1. Start with no labels. Add as concrete use cases emerge.
2. Keep cardinality below 10 per metric as a default target.
3. Initialize all label combinations you know upfront to avoid missing metrics — export 0 for known label sets.
4. Use stable label values. Avoid labels that change frequently.
5. Never include a "total" label value — rely on Prometheus sum().

Instrumentation Patterns

Online-Serving Systems (HTTP servers, APIs, databases)

Key metrics: request rate (_total), error rate, latency (histogram), in-progress (gauge).

• Count requests at completion (not start) — aligns with error and latency stats
• Always have a total requests counter alongside error counters (for ratio calculation)

Offline Processing (queues, pipelines, ETL)

Key metrics per stage: items in (_total), items out (_total), in progress (gauge), last processed timestamp (gauge), processing duration (histogram).

• Export heartbeat timestamps to detect stalled processing

Batch Jobs (cron, scheduled tasks)

Key metrics (push to Pushgateway): last success timestamp (gauge), last completion timestamp (gauge), duration (gauge — single run, not distribution).

• Batch job durations are gauges (single event), not histograms
• Jobs running more often than every 15 minutes should be converted to daemons

Libraries

Instrument transparently — users get metrics without configuration. Minimum for external resource access: request count (counter), error count (counter), latency (histogram).

Subsystem Patterns

• Logging: maintain log_messages_total{level="..."} counter per log level
• Failures: always pair failure counter with total attempts counter for ratio calculation
• Caches: cache_requests_total{result="hit|miss"}, evictions (counter), size (gauge), lookup latency (histogram). Also instrument the downstream system.

See ${CLAUDE_SKILL_DIR}/references/instrumentation.md for threadpool patterns, custom collectors, and performance tuning in hot paths.

PromQL

Rate and Increase (Counters Only)

• rate(counter[5m]) — per-second rate. Use for alerts and dashboards.
• increase(counter[5m]) — total increase. Sugar for rate() * range_seconds.
• irate(counter[5m]) — instant rate from last two samples. Only for graphing volatile counters.
• rate() first, then aggregate: sum(rate(x[5m])) — never rate(sum(x)[5m]). Rate must see individual counter resets.
• Never rate() a gauge — use deriv() or delta().

Histogram Quantiles

• histogram_quantile(0.95, rate(metric_bucket[5m])) — single histogram
• When aggregating histogram buckets, always preserve le in the by clause: histogram_quantile(0.95, sum by (job, le) (rate(metric_bucket[5m])))
• Average duration: rate(metric_sum[5m]) / rate(metric_count[5m])

PromQL Gotchas

• Staleness: most recent sample within lookback period (default 5 min). Series disappears if not scraped within that window.
• Rate window size: rate() needs at least 2 samples. Range should be at least 4x scrape interval. With 15s scrape use rate(x[5m]) or wider.
• Expensive queries: bare metric names can expand to thousands of series. Always filter or aggregate before graphing. Use recording rules for expensive expressions.

See ${CLAUDE_SKILL_DIR}/references/promql.md for data types, selectors, aggregation operators, vector matching, over-time aggregation, binary operators, and operator precedence.

Alerting Rules

Alert on symptoms (user-visible impact), not causes. Use dashboards to pinpoint causes after an alert fires.

• Online-serving — High latency, high error rate (user-facing, high in the stack)
• Offline processing — Data taking too long to get through the system
• Batch jobs — Job has not succeeded recently enough (>= 2x normal cycle)
• Capacity — Approaching resource limits that will cause outage without intervention

Only page on latency at one point in the stack — if overall user latency is fine, don't page on a slow sub-component. If an alert fires and there's nothing to do, remove it.

See ${CLAUDE_SKILL_DIR}/references/alerting-and-rules.md for alert design, naming conventions, and recording rule details.

Recording Rules

Pre-compute frequently used or expensive expressions. Format: level:metric:operations.

• Aggregate ratios correctly — aggregate numerator and denominator separately, then divide. Never average a ratio.
• Use without for aggregation — preserves all labels except those removed. Prefer over by.
• Use recording rules for expensive dashboard queries, expressions used in multiple alerts, or complex multi-step aggregations.

See ${CLAUDE_SKILL_DIR}/references/alerting-and-rules.md for full naming convention and recording rule anti-patterns.

Exporters

Write an exporter when the target system does not expose Prometheus metrics natively. For your own code, use a client library directly.

• Prefix all metrics with exporter name: haproxy_up, mysql_global_status_threads_connected
• Create fresh metric instances per scrape — do NOT use global metric variables updated each scrape (race conditions, stale labels)
• Drop pre-computed rates, min/max since start, and stddev from source systems — export raw counters and current values; let Prometheus rate() handle the rest

See ${CLAUDE_SKILL_DIR}/references/exporters.md for architecture, collectors, help strings, label rules, and push-based sources.

Application

When writing Prometheus instrumentation:

• Apply all conventions silently — don't narrate each rule.
• Choose metric types based on the decision criteria above.
• If an existing codebase contradicts a convention, follow the codebase and flag the divergence once.

When writing PromQL queries:

• Always wrap counters in rate() or increase() before further operations.
• Prefer without over by for aggregation.

When writing alerting or recording rules:

• Follow level:metric:operations naming.
• Alert on symptoms, not causes.
• Aggregate ratios correctly (numerator and denominator separately).

When reviewing Prometheus code:

• Cite the specific violation and show the fix inline.
• State what's wrong and how to fix it.

Integration

The coding skill governs workflow; this skill governs Prometheus implementation choices.