Pyroscope Troubleshooting Reference

Pyroscope Troubleshooting Reference

Complete guide for diagnosing and resolving common Pyroscope issues.

Diagnostic Commands

Pod Status

# Check all Pyroscope pods
kubectl get pods -n pyroscope -l app.kubernetes.io/name=pyroscope

# Describe specific pod
kubectl describe pod pyroscope-ingester-0 -n pyroscope

# Check pod events
kubectl get events -n pyroscope --sort-by='.lastTimestamp'

Logs

# All components
kubectl logs -n pyroscope -l app.kubernetes.io/name=pyroscope --tail=100

# Specific component
kubectl logs -n pyroscope -l app.kubernetes.io/component=ingester --tail=200
kubectl logs -n pyroscope -l app.kubernetes.io/component=distributor --tail=200
kubectl logs -n pyroscope -l app.kubernetes.io/component=querier --tail=200
kubectl logs -n pyroscope -l app.kubernetes.io/component=compactor --tail=200

# Follow logs
kubectl logs -f -n pyroscope -l app.kubernetes.io/component=ingester

# Previous container logs (after restart)
kubectl logs -n pyroscope pyroscope-ingester-0 --previous

Health Checks

# Readiness
kubectl exec -it pyroscope-0 -n pyroscope -- curl http://localhost:4040/ready

# Ring status (ingesters)
kubectl exec -it pyroscope-0 -n pyroscope -- curl http://localhost:4040/ingester/ring

# Configuration
kubectl exec -it pyroscope-0 -n pyroscope -- curl http://localhost:4040/config

# Metrics
kubectl exec -it pyroscope-0 -n pyroscope -- curl http://localhost:4040/metrics

Connectivity Tests

# Test from within cluster
kubectl run test-pyroscope --image=curlimages/curl:latest --rm -it -- \
  curl -v http://pyroscope.pyroscope:4040/ready

# Test push endpoint
kubectl run test-push --image=curlimages/curl:latest --rm -it -- \
  curl -X POST http://pyroscope.pyroscope:4040/ingest \
  -H "Content-Type: application/json" \
  -d '{"name":"test"}'

# Test from SDK perspective
kubectl run test-sdk --image=curlimages/curl:latest --rm -it -- \
  curl -v http://pyroscope-distributor.pyroscope:4040/ready

Common Issues

1. Ingester Out of Memory (OOM)

Symptoms:

• Pod restarts with OOMKilled status
• Memory usage spikes before crash
• Ingester logs show memory pressure

Diagnosis:

# Check OOM status
kubectl describe pod pyroscope-ingester-0 -n pyroscope | grep -A5 "State:"

# Check memory usage
kubectl top pods -n pyroscope -l app.kubernetes.io/component=ingester

Solutions:

# Increase memory limits
ingester:
  resources:
    limits:
      memory: 16Gi
    requests:
      memory: 8Gi

# Reduce batch size
pyroscope:
  config:
    ingester:
      max_block_duration: 30m  # Flush more frequently

2. Storage Authentication Failed

Symptoms:

• Ingester/Compactor failing to start
• "access denied" or "unauthorized" in logs
• Storage-related error messages

AWS S3:

# Verify IAM role
aws sts get-caller-identity

# Check bucket policy
aws s3 ls s3://pyroscope-bucket/

# Verify environment variables
kubectl exec -it pyroscope-ingester-0 -n pyroscope -- env | grep AWS

Azure Blob:

# Verify managed identity
az identity show --name pyroscope-identity --resource-group <rg>

# Check RBAC assignment
az role assignment list --scope /subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.Storage/storageAccounts/<storage>

# Assign if missing
az role assignment create \
  --role "Storage Blob Data Contributor" \
  --assignee-object-id <principal-id> \
  --scope <storage-scope>

# Create containers if missing
az storage container create --name pyroscope-data --account-name <storage>

GCS:

# Verify service account
kubectl get secret pyroscope-gcs-sa -n pyroscope -o yaml

# Check IAM binding
gcloud projects get-iam-policy <project> --filter="bindings.members:serviceAccount:*"

3. Profiles Not Appearing

Symptoms:

• No data in Grafana Pyroscope datasource
• SDKs appear to send successfully
• No errors in SDK logs

Diagnosis:

# Check distributor is receiving data
kubectl logs -n pyroscope -l app.kubernetes.io/component=distributor --tail=100 | grep -i "push"

# Check ingester is receiving from distributor
kubectl logs -n pyroscope -l app.kubernetes.io/component=ingester --tail=100 | grep -i "append"

# Verify ring membership
kubectl exec -it pyroscope-0 -n pyroscope -- curl http://localhost:4040/ingester/ring | jq '.shards'

Solutions:

# Verify SDK configuration
pyroscope.Start(pyroscope.Config{
    ApplicationName: "my-app",       # Must be set
    ServerAddress:   "http://pyroscope:4040", # Verify URL
    Logger:          pyroscope.StandardLogger, # Enable logging
})

# Check network policy allows traffic
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-pyroscope-ingestion
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: pyroscope
  ingress:
    - from: []
      ports:
        - port: 4040

4. Query Timeouts

Symptoms:

• Queries hang or timeout
• "context deadline exceeded" errors
• Slow flame graph rendering

Diagnosis:

# Check querier logs
kubectl logs -n pyroscope -l app.kubernetes.io/component=querier --tail=100 | grep -i "timeout\|slow"

# Check query-frontend logs
kubectl logs -n pyroscope -l app.kubernetes.io/component=query-frontend --tail=100

# Check store-gateway for historical queries
kubectl logs -n pyroscope -l app.kubernetes.io/component=store-gateway --tail=100

Solutions:

# Increase query timeout
pyroscope:
  config:
    querier:
      query_timeout: 5m
      max_concurrent: 8

    query_scheduler:
      max_outstanding_requests_per_tenant: 2048

# Scale queriers
querier:
  replicas: 5

5. High Cardinality Labels

Symptoms:

• Memory usage increases rapidly
• Slow queries
• "too many labels" errors

Diagnosis:

# Check unique label combinations
kubectl exec -it pyroscope-0 -n pyroscope -- \
  curl "http://localhost:4040/querier.v1.QuerierService/Series" \
  -H "Content-Type: application/json" \
  -d '{"matchers":[]}'

Solutions:

# Limit label cardinality
pyroscope:
  config:
    validation:
      max_label_names_per_series: 25
      max_label_name_length: 1024
      max_label_value_length: 2048

6. Ring Unhealthy

Symptoms:

• Ingesters showing as UNHEALTHY
• "ring not ready" errors
• Failed writes

Diagnosis:

# Check ring status
kubectl exec -it pyroscope-0 -n pyroscope -- curl http://localhost:4040/ingester/ring

# Check memberlist connectivity
kubectl exec -it pyroscope-0 -n pyroscope -- curl http://localhost:4040/memberlist

# Verify DNS resolution
kubectl exec -it pyroscope-0 -n pyroscope -- nslookup pyroscope-memberlist

Solutions:

# Verify memberlist configuration
pyroscope:
  config:
    memberlist:
      bind_port: 7946
      join:
        - dnssrv+pyroscope-memberlist._tcp.pyroscope.svc.cluster.local

# Check headless service exists
apiVersion: v1
kind: Service
metadata:
  name: pyroscope-memberlist
spec:
  clusterIP: None
  selector:
    app.kubernetes.io/name: pyroscope
  ports:
    - name: memberlist
      port: 7946
      protocol: UDP

7. Compactor Not Running

Symptoms:

• Blocks not being compacted
• Storage usage growing rapidly
• Old data not being deleted

Diagnosis:

# Check compactor logs
kubectl logs -n pyroscope -l app.kubernetes.io/component=compactor --tail=200

# Check compactor ring
kubectl exec -it pyroscope-compactor-0 -n pyroscope -- \
  curl http://localhost:4040/compactor/ring

Solutions:

# Verify compactor configuration
pyroscope:
  config:
    compactor:
      compaction_interval: 30m
      retention_enabled: true
      retention_delete_delay: 2h
      compaction_concurrency: 4

8. SDK Connection Refused

Symptoms:

• "connection refused" in application logs
• Profiles not being sent
• SDK timeout errors

Diagnosis:

# Test connectivity from application pod
kubectl exec -it <app-pod> -n <app-ns> -- \
  curl -v http://pyroscope.pyroscope:4040/ready

# Check service exists
kubectl get svc -n pyroscope pyroscope

Solutions:

# Verify service DNS
kubectl run test-dns --image=busybox --rm -it -- nslookup pyroscope.pyroscope.svc.cluster.local

# Check endpoint
kubectl get endpoints -n pyroscope pyroscope

9. Alloy Not Scraping

Symptoms:

• No profiles from auto-instrumented apps
• Alloy running but not collecting data
• Annotations present but ignored

Diagnosis:

# Check Alloy logs
kubectl logs -n pyroscope -l app.kubernetes.io/name=alloy --tail=200

# Verify targets
kubectl exec -it <alloy-pod> -n pyroscope -- \
  wget -qO- http://localhost:12345/agent/api/v1/targets

Solutions:

# Verify pod annotations
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    metadata:
      annotations:
        profiles.grafana.com/cpu.scrape: "true"
        profiles.grafana.com/cpu.port: "8080"

# Check Alloy configuration
alloy:
  alloy:
    configMap:
      content: |
        discovery.kubernetes "pods" {
          role = "pod"
        }
        pyroscope.scrape "default" {
          targets = discovery.kubernetes.pods.targets
          forward_to = [pyroscope.write.default.receiver]
        }

10. Store-Gateway Slow Sync

Symptoms:

• Historical queries slow
• Store-gateway high CPU during sync
• "bucket sync" messages in logs

Diagnosis:

# Check sync status
kubectl logs -n pyroscope -l app.kubernetes.io/component=store-gateway --tail=200 | grep -i "sync"

# Check block count
kubectl exec -it pyroscope-store-gateway-0 -n pyroscope -- \
  curl http://localhost:4040/store-gateway/blocks

Solutions:

# Tune sync interval
pyroscope:
  config:
    store_gateway:
      sync_interval: 30m  # Increase if too frequent
      ignore_blocks_within: 6h  # Skip recent blocks

Performance Tuning

Ingestion Optimization

# High throughput configuration
pyroscope:
  config:
    distributor:
      push_timeout: 10s
      ingestion_tenant_shard_size: 8

    ingester:
      max_block_duration: 1h
      flush_check_period: 30s

Query Optimization

# Faster queries
pyroscope:
  config:
    querier:
      max_concurrent: 16
      query_store_after: 2h

    query_frontend:
      max_outstanding_per_tenant: 400

Storage Optimization

# Efficient storage
pyroscope:
  config:
    compactor:
      compaction_concurrency: 8
      block_ranges:
        - 1h
        - 4h
        - 12h

Metrics to Monitor

Key Metrics

Prometheus Rules

groups:
  - name: pyroscope
    rules:
      - alert: PyroscopeIngesterOOM
        expr: |
          container_memory_working_set_bytes{container="ingester"}
          / container_spec_memory_limit_bytes{container="ingester"} > 0.9
        for: 5m
        labels:
          severity: warning

      - alert: PyroscopeCompactorNotRunning
        expr: |
          increase(pyroscope_compactor_runs_completed_total[2h]) == 0
        for: 10m
        labels:
          severity: warning

      - alert: PyroscopeRingUnhealthy
        expr: |
          pyroscope_ring_members{state="UNHEALTHY"} > 0
        for: 5m
        labels:
          severity: critical

Log Analysis

Common Log Patterns

Successful ingestion:

level=info component=distributor msg="pushed profiles" tenant=anonymous count=100

Storage errors:

level=error component=ingester msg="failed to upload block" err="access denied"

Ring issues:

level=warn component=ring msg="instance not found in ring" instance=pyroscope-ingester-0

Query issues:

level=error component=querier msg="query failed" err="context deadline exceeded"

Grep Commands

# Find errors
kubectl logs -n pyroscope -l app.kubernetes.io/name=pyroscope | grep -i "error\|fail\|panic"

# Find storage issues
kubectl logs -n pyroscope -l app.kubernetes.io/component=ingester | grep -i "storage\|upload\|s3\|azure\|gcs"

# Find ring issues
kubectl logs -n pyroscope -l app.kubernetes.io/name=pyroscope | grep -i "ring\|memberlist"

# Find query issues
kubectl logs -n pyroscope -l app.kubernetes.io/component=querier | grep -i "query\|timeout"

Recovery Procedures

Restart Components

# Restart specific component
kubectl rollout restart deployment/pyroscope-distributor -n pyroscope
kubectl rollout restart statefulset/pyroscope-ingester -n pyroscope

# Restart all
kubectl rollout restart deployment -n pyroscope
kubectl rollout restart statefulset -n pyroscope

Clear Stuck State

# Delete PVC if corrupted (data loss!)
kubectl delete pvc pyroscope-ingester-data-pyroscope-ingester-0 -n pyroscope

# Force delete stuck pod
kubectl delete pod pyroscope-ingester-0 -n pyroscope --force --grace-period=0

Scale Down/Up

# Scale down
kubectl scale statefulset pyroscope-ingester --replicas=0 -n pyroscope

# Scale up
kubectl scale statefulset pyroscope-ingester --replicas=3 -n pyroscope