# Openshift Eng AI Helpers Must Gather Analyzer > Must-Gather Analyzer Skill - Skill: `tomevault-io/openshift-eng-ai-helpers-must-gather-analyzer` (Agent Skill, multi-file: 2 files) - Install (CLI): `npx skillmds@latest add tomevault-io/openshift-eng-ai-helpers-must-gather-analyzer` - Raw SKILL.md: https://api.skillmd.com/api/skills/tomevault-io/openshift-eng-ai-helpers-must-gather-analyzer/raw - Safety review: pending (external: skill-scanner PASS, skillspector PASS) - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: tomevault-io (https://skillmd.com/u/tomevault-io) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/tomevault-io/openshift-eng-ai-helpers-must-gather-analyzer --- # Must-Gather Analyzer Skill Comprehensive analysis of OpenShift must-gather diagnostic data with helper scripts that parse YAML and display output in `oc`-like format. ## Overview This skill provides analysis for: - **ClusterVersion**: Current version, update status, and capabilities - **Cluster Operators**: Status, degradation, and availability - **Pods**: Health, restarts, crashes, and failures across namespaces - **Nodes**: Conditions, capacity, and readiness - **Network**: OVN/SDN diagnostics and connectivity - **Events**: Warning and error events across namespaces - **etcd**: Cluster health, member status, and quorum - **Storage**: PersistentVolumes and PersistentVolumeClaims status ## Must-Gather Directory Structure **Important**: Must-gather data is contained in a subdirectory with a long hash name: ``` must-gather/ └── registry-ci-openshift-org-origin-...-sha256-/ ├── cluster-scoped-resources/ │ ├── config.openshift.io/clusteroperators/ │ └── core/nodes/ ├── namespaces/ │ └── / │ └── pods/ │ └── / │ └── .yaml └── network_logs/ ``` The analysis scripts expect the path to the **subdirectory** (the one with the hash), not the root must-gather folder. ## Instructions ### 1. Get Must-Gather Path Ask the user for the must-gather directory path if not already provided. - If they provide the root directory, look for the subdirectory with the hash name - The correct path contains `cluster-scoped-resources/` and `namespaces/` directories ### 2. Choose Analysis Type Based on user's request, run the appropriate helper script: #### ClusterVersion Analysis ```bash ./scripts/analyze_clusterversion.py ``` Shows cluster version information similar to `oc get clusterversion`: - Current version and update status - Progressing state - Available updates - Version conditions - Enabled capabilities - Update history #### Cluster Operators Analysis ```bash ./scripts/analyze_clusteroperators.py ``` Shows cluster operator status similar to `oc get clusteroperators`: - Available, Progressing, Degraded conditions - Version information - Time since condition change - Detailed messages for operators with issues #### Pods Analysis ```bash # All namespaces ./scripts/analyze_pods.py # Specific namespace ./scripts/analyze_pods.py --namespace # Show only problematic pods ./scripts/analyze_pods.py --problems-only ``` Shows pod status similar to `oc get pods -A`: - Ready/Total containers - Status (Running, Pending, CrashLoopBackOff, etc.) - Restart counts - Age - Categorized issues (crashlooping, pending, failed) #### Nodes Analysis ```bash ./scripts/analyze_nodes.py # Show only nodes with issues ./scripts/analyze_nodes.py --problems-only ``` Shows node status similar to `oc get nodes`: - Ready status - Roles (master, worker) - Age - Kubernetes version - Node conditions (DiskPressure, MemoryPressure, etc.) - Capacity and allocatable resources #### Network Analysis ```bash ./scripts/analyze_network.py ``` Shows network health: - Network type (OVN-Kubernetes, OpenShift SDN) - Network operator status - OVN pod health - PodNetworkConnectivityCheck results - Network-related issues #### Events Analysis ```bash # Recent events (last 100) ./scripts/analyze_events.py # Warning events only ./scripts/analyze_events.py --type Warning # Events in specific namespace ./scripts/analyze_events.py --namespace openshift-etcd # Show last 50 events ./scripts/analyze_events.py --count 50 ``` Shows cluster events: - Event type (Warning, Normal) - Last seen timestamp - Reason and message - Affected object - Event count #### etcd Analysis ```bash ./scripts/analyze_etcd.py ``` Shows etcd cluster health: - Member health status - Member list with IDs and URLs - Endpoint status (leader, version, DB size) - Quorum status - Cluster summary #### Storage Analysis ```bash # All PVs and PVCs ./scripts/analyze_pvs.py # PVCs in specific namespace ./scripts/analyze_pvs.py --namespace openshift-monitoring ``` Shows storage resources: - PersistentVolumes (capacity, status, claims) - PersistentVolumeClaims (binding, capacity) - Storage classes - Pending/unbound volumes #### Monitoring Analysis ```bash # All alerts. ./scripts/analyze_prometheus.py # Alerts in specific namespace ./scripts/analyze_prometheus.py --namespace openshift-monitoring ``` Shows monitoring information: - Alerts (state, namespace, name, active since, labels) - Total of pending/firing alerts ### 3. Interpret and Report After running the scripts: 1. Review the summary statistics 2. Focus on items flagged with issues 3. Provide actionable insights and next steps 4. Suggest log analysis for specific components if needed 5. Cross-reference issues (e.g., degraded operator → failing pods → node issues) ## Output Format All scripts provide: - **Summary Section**: High-level statistics with emoji indicators - **Table View**: `oc`-like formatted output - **Issues Section**: Detailed breakdown of problems Example summary format: ``` ================================================================================ SUMMARY: 25/28 operators healthy ⚠️ 3 operators with issues 🔄 1 progressing ❌ 2 degraded ================================================================================ ``` ## Helper Scripts Reference ### scripts/analyze_clusterversion.py Parses: `cluster-scoped-resources/config.openshift.io/clusterversions/version.yaml` Output: ClusterVersion table with detailed version info, conditions, and capabilities ### scripts/analyze_clusteroperators.py Parses: `cluster-scoped-resources/config.openshift.io/clusteroperators/` Output: ClusterOperator status table with conditions ### scripts/analyze_pods.py Parses: `namespaces/*/pods/*/*.yaml` (individual pod directories) Output: Pod status table with issues categorized ### scripts/analyze_nodes.py Parses: `cluster-scoped-resources/core/nodes/` Output: Node status table with conditions and capacity ### scripts/analyze_network.py Parses: `network_logs/`, network operator, OVN resources Output: Network health summary and diagnostics ### scripts/analyze_events.py Parses: `namespaces/*/core/events.yaml` Output: Event table sorted by last occurrence ### scripts/analyze_etcd.py Parses: `etcd_info/` (endpoint_health.json, member_list.json, endpoint_status.json) Output: etcd cluster health and member status ### scripts/analyze_pvs.py Parses: `cluster-scoped-resources/core/persistentvolumes/`, `namespaces/*/core/persistentvolumeclaims.yaml` Output: PV and PVC status tables ## Tips for Analysis 1. **Start with Cluster Operators**: They often reveal system-wide issues 2. **Check Timing**: Look at "SINCE" columns to understand when issues started 3. **Follow Dependencies**: Degraded operator → check its namespace pods → check hosting nodes 4. **Look for Patterns**: Multiple pods failing on same node suggests node issue 5. **Cross-reference**: Use multiple scripts together for complete picture ## Common Scenarios ### "Why is my cluster degraded?" 1. Run `analyze_clusteroperators.py` - identify degraded operators 2. Run `analyze_pods.py --namespace ` - check operator pods 3. Run `analyze_nodes.py` - verify node health ### "Pods keep crashing" 1. Run `analyze_pods.py --problems-only` - find crashlooping pods 2. Check which nodes they're on 3. Run `analyze_nodes.py` - verify node conditions 4. Suggest checking pod logs in must-gather data ### "Network connectivity issues" 1. Run `analyze_network.py` - check network health 2. Run `analyze_pods.py --namespace openshift-ovn-kubernetes` 3. Check PodNetworkConnectivityCheck results ## Next Steps After Analysis Based on findings, suggest: - Examining specific pod logs in `namespaces//pods///logs/` - Reviewing events in `namespaces//core/events.yaml` - Checking audit logs in `audit_logs/` - Analyzing metrics data if available - Looking at host service logs in `host_service_logs/` --- > Converted and distributed by [TomeVault](https://tomevault.io/claim/openshift-eng) — claim your Tome and manage your conversions.