tjhop Troubleshooting Missing Data
Work out why an expected metric or series is absent: the target is not being scraped, the target no longer exposes it, the labels changed, or the query itself misses data that exists. Narrow down where the pipeline breaks.
Getting oriented
- list_targets shows scrape target health, the lastError for failing scrapes, and the labels applied at scrape time.
- label_values on name (optionally with matchers) confirms whether a metric name exists at all right now.
- targets_metadata lists which metrics a given target actually exposes.
- config shows the scrape configuration: job definitions, relabeling, and intervals.
- Prometheus's own scrape metrics explain silent drops: nonzero rates on the prometheus_target_scrapes_* rejection counters (sample_out_of_order, sample_out_of_bounds, duplicate_timestamp, exceeded_sample_limit) mean samples arrived but were rejected at ingestion.
Topics worth exploring
Treat these as starting points and follow where the data stops:
- Is the target up and scraped? query up{job=""} and check list_targets for down or missing targets and their scrape errors.
- Does the metric exist under a different shape? Search broadly, e.g. series with a matcher like {name=~".."} -- renames and label changes are common after upgrades.
- Did it exist before? range_query the series over a wide window to find exactly when it disappeared, then correlate with deploys or config changes.
- Is relabeling dropping it? Look through relabel_configs and metric_relabel_configs in config for rules that drop the target, metric, or label you expected.
- Is it the query, not the data? Staleness handling (series go stale after 5m without samples), a too-narrow time window, or an over-strict matcher can make live data look absent.
Reporting findings
State where the data stops existing (target, scrape, relabeling, or query), since when, and what change would restore it.