Kubeblocks Setup Monitoring

Set Up Monitoring for KubeBlocks Clusters

Overview

KubeBlocks database clusters export Prometheus-compatible metrics. This skill sets up a monitoring stack using Prometheus Operator and Grafana to collect and visualize database metrics.

Official docs: https://kubeblocks.io/docs/preview/user_docs/observability/monitor-database

Workflow

- [ ] Step 1: Install Prometheus stack
- [ ] Step 2: Create PodMonitor for database clusters
- [ ] Step 3: Verify metrics collection
- [ ] Step 4: Access Grafana dashboards

Step 1: Install Prometheus Stack

Install the kube-prometheus-stack which includes Prometheus Operator, Prometheus, Grafana, and Alertmanager:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install prometheus prometheus-community/kube-prometheus-stack \
  -n monitoring --create-namespace \
  --set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false \
  --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false

The NilUsesHelmValues=false settings ensure Prometheus discovers PodMonitors/ServiceMonitors from all namespaces, not just those with the release: prometheus label.

Wait for pods to be ready:

kubectl get pods -n monitoring -w

Expected pods:

• prometheus-kube-prometheus-operator-*
• prometheus-prometheus-kube-prometheus-prometheus-0
• prometheus-grafana-*
• alertmanager-prometheus-kube-prometheus-alertmanager-0

Step 2: Create PodMonitor

Create a PodMonitor to tell Prometheus where to scrape database metrics:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: kb-database-monitor
  namespace: monitoring
  labels:
    release: prometheus
spec:
  namespaceSelector:
    matchNames:
    - <ns>    # namespace where your database cluster runs
  selector:
    matchLabels:
      app.kubernetes.io/managed-by: kubeblocks
  podMetricsEndpoints:
  - port: http-metrics
    path: /metrics
    interval: 30s

Apply it:

kubectl apply -f podmonitor.yaml

Monitor All Namespaces

To monitor KubeBlocks clusters across all namespaces:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: kb-database-monitor-all
  namespace: monitoring
  labels:
    release: prometheus
spec:
  namespaceSelector:
    any: true
  selector:
    matchLabels:
      app.kubernetes.io/managed-by: kubeblocks
  podMetricsEndpoints:
  - port: http-metrics
    path: /metrics
    interval: 30s

Per-Component PodMonitor

For more granular control, target a specific component:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: kb-mysql-monitor
  namespace: monitoring
  labels:
    release: prometheus
spec:
  namespaceSelector:
    matchNames:
    - <ns>
  selector:
    matchLabels:
      app.kubernetes.io/managed-by: kubeblocks
      apps.kubeblocks.io/component-name: <component>
      app.kubernetes.io/instance: <cluster>
  podMetricsEndpoints:
  - port: http-metrics
    path: /metrics
    interval: 15s

Step 3: Verify Metrics Collection

Check PodMonitor is Discovered

kubectl get podmonitor -n monitoring

Query Prometheus Directly

Port-forward to Prometheus:

kubectl port-forward -n monitoring svc/prometheus-kube-prometheus-prometheus 9090:9090

Then open http://localhost:9090 in a browser, or test with curl:

# Check targets (should show your database pods)
curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | select(.labels.app_kubernetes_io_managed_by == "kubeblocks") | {instance: .labels.instance, health: .health}'

# Query a sample metric
curl -s 'http://localhost:9090/api/v1/query?query=up{app_kubernetes_io_managed_by="kubeblocks"}' | jq '.data.result'

Common KubeBlocks Metrics

Step 4: Access Grafana

Port-Forward to Grafana

kubectl port-forward -n monitoring svc/prometheus-grafana 3000:80

Open http://localhost:3000 in a browser.

Default Credentials

Username: admin
Password: prom-operator

Get the password programmatically:

kubectl get secret -n monitoring prometheus-grafana \
  -o jsonpath='{.data.admin-password}' | base64 -d

Import Database Dashboards

Grafana comes with some built-in dashboards. For database-specific dashboards, import from Grafana.com:

To import: Grafana → Dashboards → Import → Enter the ID → Select Prometheus data source → Import.

Troubleshooting

No metrics appearing in Prometheus:

• Check PodMonitor labels match: release: prometheus must be present
• Verify the metrics port name: kubectl get pods <pod> -n <ns> -o jsonpath='{.spec.containers[*].ports[*]}'
• Some addons use different port names; check with: kubectl describe pod <pod> -n <ns> | grep -A 2 "Port:"

Targets show as DOWN in Prometheus:

• Check pod network connectivity from the monitoring namespace
• Verify the metrics endpoint responds: kubectl exec -it <pod> -n <ns> -- curl -s localhost:9104/metrics | head

Grafana cannot reach Prometheus:

• Verify Prometheus service: kubectl get svc -n monitoring | grep prometheus
• The default data source URL is http://prometheus-kube-prometheus-prometheus.monitoring:9090

Additional Reference

For per-addon exporter details (port names, port numbers, metrics paths), custom alerting rule examples (MySQL, PostgreSQL, Redis), and extended Grafana dashboard IDs, see reference.md.