dvcrn Troubleshoot KubeBlocks Clusters
Overview
This skill helps diagnose and fix common issues with KubeBlocks-managed database clusters. Follow the diagnostic flowchart and sections below to systematically identify root causes. This skill guides the agent through diagnostic steps; it does not perform actions.
Quick Diagnostic Flowchart
User reports cluster problem
│
├─ Is the KubeBlocks operator healthy?
│ └─ kubectl -n kb-system get pods
│ ├─ Pods not Running → Operator Issues (Section 5)
│ └─ Pods Running → continue
│
├─ Is the cluster in an abnormal state?
│ └─ kubectl get cluster <name> -n <ns>
│ ├─ Phase: Creating/Updating/Abnormal/Deleting → Cluster Status Issues (Section 1)
│ └─ Phase: Running → continue
│
├─ Is there a failed OpsRequest?
│ └─ kubectl get opsrequest -n <ns>
│ ├─ Failed/Running (stuck) → OpsRequest Failures (Section 3)
│ └─ Succeeded → continue
│
└─ Are pods crashing?
└─ kubectl get pods -n <ns> -l app.kubernetes.io/instance=<cluster>
├─ CrashLoopBackOff/ImagePullBackOff/Pending → Pod Issues (Section 2)
└─ All Running → check pod logs and events for application errors
1. Cluster Status Issues
Cluster stuck in Creating
| Cause | How to verify | Action |
|---|---|---|
| Addon not installed | kubectl get addon |
Install the required addon (e.g. kbcli addon enable mysql) |
| Insufficient resources | kubectl describe pod <pending-pod> → Events |
Increase node resources or reduce cluster requests |
| Image pull errors | kubectl describe pod → ImagePullBackOff |
Fix registry access, image name, or imagePullSecrets |
Cluster stuck in Updating
- Check if all pods are
Running:kubectl get pods -n <ns> -l app.kubernetes.io/instance=<cluster> - Check pod logs for errors:
kubectl logs <pod> -c <container> - Verify pod roles:
kubectl get po <pod> -L kubeblocks.io/role - Check for failed OpsRequest:
kubectl get opsrequest -n <ns>andkubectl describe opsrequest <name> - Ensure container image in
status.containerStatusesmatchesspec.containers.image; mismatches can block updates
Cluster in Abnormal
Usually caused by pod failures or storage issues. Check pod status and events (see Pod Issues below), then kubectl describe cluster <name> -n <ns> for conditions.
Cluster stuck in Deleting
If KubeBlocks logs show has no pods to running the pre-terminate action, the cluster cannot run the pre-terminate lifecycle. To skip:
kubectl annotate component <COMPONENT_NAME> -n <ns> apps.kubeblocks.io/skip-pre-terminate-action=true
2. Pod Issues
CrashLoopBackOff
- Check logs:
kubectl logs <pod> -c <container> --previous - Describe pod:
kubectl describe pod <pod> -n <ns>(check Events) - Common causes: Wrong config, bad credentials, storage mount failure, OOM, database init failure
ImagePullBackOff
- Describe pod:
kubectl describe pod <pod>→ Events show pull error - Common causes: Private registry without
imagePullSecrets, wrong image name/tag, network/registry unreachable
Pending
- Describe pod:
kubectl describe pod <pod>→ Events - Common causes: Insufficient CPU/memory, no nodes matching nodeSelector/affinity, PVC pending (StorageClass or capacity)
3. OpsRequest Failures
How to check failed OpsRequest
kubectl get opsrequest -n <namespace>
kubectl describe opsrequest <ops-name> -n <namespace>
Check status.conditions and Events for failure reason.
Common OpsRequest failure reasons
- Resource constraints (scaling beyond available capacity)
- Preconditions not met (e.g. cluster not Running)
- Timeout or step failure during the operation
How to cancel a stuck OpsRequest
Only VerticalScaling and HorizontalScaling OpsRequests in Running state can be cancelled:
kubectl patch opsrequest <OPSREQUEST_NAME> -n <ns> -p '{"spec":{"cancel":true}}' --type=merge
4. Operator Issues
KubeBlocks operator pod not running
kubectl -n kb-system get pods
kubectl -n kb-system describe pod <kubeblocks-pod>
CRD version mismatch
If installing on K8s ≤ 1.23, you may see unknown field "x-kubernetes-validations". Apply CRDs with --validate=false (see official docs).
Check operator logs
kubectl -n kb-system logs -l app.kubernetes.io/name=kubeblocks --tail=100 -f
# or
kubectl -n kb-system logs deployments/kubeblocks -f
ComponentDefinition status Unavailable
If kubectl get componentdefinition shows Unavailable (e.g. "immutable fields can't be updated"):
kubectl annotate componentdefinition <NAME> apps.kubeblocks.io/skip-immutable-check=true
5. Useful Commands Reference
| Purpose | Command |
|---|---|
| Operator health | kubectl -n kb-system get pods |
| Cluster status | kubectl get cluster -A |
| Cluster details | kubectl describe cluster <name> -n <ns> |
| OpsRequest status | kubectl get opsrequest -n <ns> |
| Pod status | kubectl get pods -n <ns> -l app.kubernetes.io/instance=<cluster> |
| Pod logs | kubectl logs <pod> -c <container> -n <ns> |
| Pod events | kubectl describe pod <pod> -n <ns> |
| Operator logs | kubectl -n kb-system logs -l app.kubernetes.io/name=kubeblocks --tail=100 |
| Cluster resources | kubectl get cmp,its,po -l app.kubernetes.io/instance=<cluster> -n <ns> |
| Generate report | kbcli report cluster <name> --with-logs --mask |
6. Official Troubleshooting Docs
| Resource | URL |
|---|---|
| FAQs (cluster exception) | https://kubeblocks.io/docs/preview/user_docs/troubleshooting/handle-a-cluster-exception |
| Known Issues | https://kubeblocks.io/docs/preview/user_docs/troubleshooting/known-issues |
| Full doc index | https://kubeblocks.io/llms-full.txt |
Known issues to check
- PostgreSQL password with special chars (v0.9.4 and before, v1.0.0): Upgrade to v1.0.1-beta.6+ or set
symbolCharactersinpasswordConfigto avoid YAML parsing errors. - Excessive secrets (KubeBlocks v1.0.0 on K8s ≤ 1.24): Upgrade to v1.0.1-beta.3+.