How do I install the Check Ceph Health skill?

Run npx skillmds add yaacov/check-ceph-health in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

What does the Check Ceph Health skill do?

Check Ceph storage health on OpenShift OCS/ODF clusters. Use when PVCs are stuck in Pending, storage provisioning fails, Ceph is degraded, OSDs are full, or cluster storage needs diagnosis. It is listed under Coding & Dev Tools on SkillMD.

Is Check Ceph Health safe to use?

This skill has not completed SkillMD's automated safety review yet. Capability flags: makes network calls, reads secrets. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Check Ceph Health?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Check Ceph Health free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Check Ceph Health?

yaacov (@yaacov) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Check Ceph Health by yaacov — Agent Skill

Name: Check Ceph Health
Rating: 4.2 (5 reviews)
Author: yaacov

Check Ceph Health

Use this guide to diagnose and remediate Ceph storage issues on OpenShift clusters running OCS/ODF (OpenShift Data Foundation).

Required CLI Tools

This skill requires:

oc metrics (kubectl-metrics) -- for Ceph metrics (health, capacity, OSD, PG)
oc debug-queries (kubectl-debug-queries) -- for listing resources, logs, events

If any tool is missing, install with the secure version-pinned installer:

curl -sSLO https://raw.githubusercontent.com/kubev2v/mtv-skills/main/tools/install-tools.sh
curl -sSL  https://raw.githubusercontent.com/kubev2v/mtv-skills/main/SHA256SUMS | shasum -a 256 --check --ignore-missing
bash install-tools.sh kubectl-metrics kubectl-debug-queries && rm install-tools.sh

Using `--query` for Filtering

Use --query to filter, sort, and project results server-side. Use pipe output to jq, grep, or other post-processing tools only when --query cannot express what you need.

The --query flag accepts TSL (Tree Search Language) with four optional clauses:

[select <field>, ...] [where <condition>] [order by <field> [asc|desc]] [limit N]

Note: select only affects table output (the default). With --output json, all fields are always returned regardless of select.

Operators: =, !=, <, >, <=, >=, like (% wildcard), ilike (case-insensitive), ~= (regex), and, or, not, in [...], between X and Y.

Before writing queries, discover actual field names with --output json:

oc debug-queries list --resource pods --namespace openshift-storage --limit 2 --output json

1. Ceph Cluster Health

Quick health status via metrics

oc metrics query --query "ceph_health_status"

Health values: 0=OK, 1=WARN, 2=ERR.

Capacity overview

oc metrics query --query "ceph_cluster_total_bytes"
oc metrics query --query "ceph_cluster_total_used_bytes"
oc metrics query --query "ceph_cluster_total_used_bytes / ceph_cluster_total_bytes * 100"

CephCluster CR status

oc debug-queries get --resource cephcluster --namespace openshift-storage --output json

Health states:

HEALTH_OK -- cluster is healthy
HEALTH_WARN -- degraded but functional (backfillfull, nearfull, degraded PGs)
HEALTH_ERR -- critical, writes may be blocked (full OSDs, too few OSDs, down PGs)

2. OSD Status

OSD metrics

oc metrics query --query "ceph_osd_stat_bytes"
oc metrics query --query "ceph_osd_stat_bytes_used"
oc metrics query --query "rate(ceph_osd_op_latency_sum[5m]) / rate(ceph_osd_op_latency_count[5m])"

OSD pods

oc debug-queries list --resource pods --namespace openshift-storage --selector "app=rook-ceph-osd"
oc debug-queries list --resource pods --namespace openshift-storage --query "where name ~= '.*osd-prepare.*'"

OSD backing PVCs

oc debug-queries list --resource pvc --namespace openshift-storage --selector "app=rook-ceph-osd"

3. Placement Group Health

oc metrics query --query "ceph_pg_total"
oc metrics query --query "ceph_pg_active"
oc metrics query --query "ceph_pg_degraded"

4. Pool Statistics

oc metrics query --query "ceph_pool_percent_used * 100"
oc metrics query --query "rate(ceph_pool_rd[5m])"
oc metrics query --query "rate(ceph_pool_wr[5m])"
oc metrics query --query "ceph_pool_stored"
oc metrics query --query "ceph_pool_max_avail"

5. CSI Provisioner Pods

PVC provisioning is handled by CSI driver pods. If these are unhealthy, no volumes can be created.

oc debug-queries list --resource pods --namespace openshift-storage --query "where name ~= '.*rbd.*ctrlplugin.*'"
oc debug-queries list --resource pods --namespace openshift-storage --query "where name ~= '.*cephfs.*ctrlplugin.*'"
oc debug-queries list --resource pods --namespace openshift-storage --query "where name ~= '.*rbd.*nodeplugin.*'"

Check CSI provisioner logs:

oc debug-queries logs --name <rbd-ctrlplugin-pod> --namespace openshift-storage --container csi-rbdplugin --tail 50

6. PVC and PV Diagnosis

oc debug-queries list --resource pvc --all-namespaces --query "where status.phase = 'Pending'"
oc debug-queries get --resource pvc --name <pvc-name> --namespace <namespace>
oc debug-queries list --resource pv --all-namespaces --query "where status.phase = 'Released'"
oc debug-queries list --resource storageclass --all-namespaces

7. Common Problems and Remediation

OSDs Full (HEALTH_ERR: full osd(s))

Symptoms: PVCs stuck in Pending, provisioning errors with DeadlineExceeded or operation already exists.

Diagnosis:

oc metrics query --query "ceph_health_status"
oc metrics query --query "ceph_osd_stat_bytes_used / ceph_osd_stat_bytes * 100"
oc debug-queries get --resource cephcluster --namespace openshift-storage --output json

Look for OSD_FULL and POOL_FULL messages in the CephCluster status.

Remediation: See "Requires Shell" section below for oc delete pv and ceph osd set-full-ratio.

OSDs Nearfull / Backfillfull (HEALTH_WARN)

Symptoms: Cluster functional but approaching full. Warnings about nearfull or backfillfull OSDs.

Remediation:

Clean up unused PVCs and Released PVs
Delete completed migration data no longer needed
Plan capacity expansion before reaching full threshold (85%)

Degraded PGs

Symptoms: HEALTH_WARN with messages about degraded or undersized placement groups.

Diagnosis:

oc metrics query --query "ceph_pg_degraded"
oc metrics query --query "ceph_pg_total - ceph_pg_active"
oc debug-queries events --namespace openshift-storage --query "where type = 'Warning'"

Remediation:

If an OSD is down, check the OSD pod and its node
If a node is down, Ceph will self-heal once the node returns
If an OSD is permanently lost, Ceph will rebalance automatically (may take time)

CSI Provisioner Not Responding

Symptoms: PVC events say "waiting for external provisioner" but no ProvisioningFailed errors.

Diagnosis:

oc debug-queries list --resource pods --namespace openshift-storage --query "where name ~= '.*ctrlplugin.*'"
oc debug-queries logs --name <rbd-ctrlplugin-pod> --namespace openshift-storage --container csi-rbdplugin --tail 100

Remediation:

Restart the CSI controller pod if it's stuck
Check if the Ceph cluster is reachable from the CSI pod
Verify the StorageClass references a valid pool and secret

Pools Full but OSDs Not Full

Symptoms: POOL_FULL warning but individual OSDs have space.

Diagnosis:

oc metrics query --query "ceph_pool_percent_used * 100"
oc metrics query --query "ceph_osd_stat_bytes_used / ceph_osd_stat_bytes * 100"

Remediation:

A pool may have a quota set -- check and raise it
Rebalance may be needed if data is unevenly distributed

8. Operator Health

oc debug-queries list --resource pods --namespace openshift-storage --query "where name ~= '.*ocs-operator.*|.*odf-operator.*|.*rook-ceph-operator.*'"
oc debug-queries logs --name deployment/rook-ceph-operator --namespace openshift-storage --tail 50

Check for high restart counts:

oc metrics query --query "topk(10, sort_desc(kube_pod_container_status_restarts_total))"

9. Preventive Checks

Run these periodically to avoid surprise outages:

oc metrics query --query "ceph_cluster_total_used_bytes / ceph_cluster_total_bytes * 100"
oc debug-queries list --resource pv --all-namespaces --query "where status.phase = 'Released'"
oc debug-queries list --resource pvc --all-namespaces --query "where status.phase = 'Pending'"

Act when usage exceeds 70% -- start cleaning up or expanding capacity before hitting the 85% full threshold.

Requires Shell

These remediation operations require shell access:

Delete Released PVs to reclaim space

oc get pv --field-selector status.phase=Released
oc delete pv <released-pv-names>

Temporarily raise the full ratio (when Ceph is blocking all writes)

MON_POD=$(oc -n openshift-storage get pods -l app=rook-ceph-mon -o jsonpath='{.items[0].metadata.name}')
MON_ADDR=$(oc -n openshift-storage get pod $MON_POD -o jsonpath='{.spec.containers[0].env[?(@.name=="ROOK_CEPH_MON_HOST")].value}' | sed 's/\[//;s/\]//')

# Raise to 0.92 to unblock writes temporarily
oc -n openshift-storage exec $MON_POD -c mon -- \
  ceph -m $MON_ADDR --keyring /etc/ceph/keyring-store/keyring \
  osd set-full-ratio 0.92

# After space is freed, reset to default
oc -n openshift-storage exec $MON_POD -c mon -- \
  ceph -m $MON_ADDR --keyring /etc/ceph/keyring-store/keyring \
  osd set-full-ratio 0.85

Self-Learning Rule

When you need to discover available flags or verify syntax:

oc debug-queries list --help
oc debug-queries logs --help
oc metrics query --help

Check Ceph Health

File contents

Check Ceph Health

Required CLI Tools

Using `--query` for Filtering

1. Ceph Cluster Health

Quick health status via metrics

Capacity overview

CephCluster CR status

2. OSD Status

OSD metrics

OSD pods

OSD backing PVCs

3. Placement Group Health

4. Pool Statistics

5. CSI Provisioner Pods

6. PVC and PV Diagnosis

7. Common Problems and Remediation

OSDs Full (HEALTH_ERR: full osd(s))

OSDs Nearfull / Backfillfull (HEALTH_WARN)

Degraded PGs

CSI Provisioner Not Responding

Pools Full but OSDs Not Full

8. Operator Health

9. Preventive Checks

Requires Shell

Delete Released PVs to reclaim space

Temporarily raise the full ratio (when Ceph is blocking all writes)

Self-Learning Rule

Frequently asked questions

Check Ceph Health

File contents

Check Ceph Health

Required CLI Tools

Using --query for Filtering

1. Ceph Cluster Health

Quick health status via metrics

Capacity overview

CephCluster CR status

2. OSD Status

OSD metrics

OSD pods

OSD backing PVCs

3. Placement Group Health

4. Pool Statistics

5. CSI Provisioner Pods

6. PVC and PV Diagnosis

7. Common Problems and Remediation

OSDs Full (HEALTH_ERR: full osd(s))

OSDs Nearfull / Backfillfull (HEALTH_WARN)

Degraded PGs

CSI Provisioner Not Responding

Pools Full but OSDs Not Full

8. Operator Health

9. Preventive Checks

Requires Shell

Delete Released PVs to reclaim space

Temporarily raise the full ratio (when Ceph is blocking all writes)

Self-Learning Rule

Related skills

Frequently asked questions

How do I install the Check Ceph Health skill?

What does the Check Ceph Health skill do?

Is Check Ceph Health safe to use?

Which AI agents work with Check Ceph Health?

Is Check Ceph Health free to use?

Who published Check Ceph Health?

Using `--query` for Filtering