# Marknygaard Home Ops Home Ops Debug > home-ops-debug - Skill: `tomevault-io/marknygaard-home-ops-home-ops-debug` (Agent Skill, multi-file: 2 files) - Install (CLI): `npx skillmds@latest add tomevault-io/marknygaard-home-ops-home-ops-debug` - Raw SKILL.md: https://api.skillmd.com/api/skills/tomevault-io/marknygaard-home-ops-home-ops-debug/raw - Safety review: pending (external: skill-scanner PASS, skillspector PASS) - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: tomevault-io (https://skillmd.com/u/tomevault-io) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/tomevault-io/marknygaard-home-ops-home-ops-debug --- # home-ops-debug Diagnoses issues in the home-ops Kubernetes cluster using the Flux MCP and Grafana MCP tools. Follows a structured triage workflow to identify root causes efficiently. ## Cluster constants | Property | Value | |----------|-------| | Flux namespace | `flux-system` | | App namespaces | `cert-manager`, `database`, `default`, `flux-system`, `kube-system`, `media`, `monitoring`, `network`, `security`, `storage`, `system-upgrade` | | Prometheus datasource | Discover UID via `list_datasources` (type: `prometheus`) | | Loki datasource | Discover UID via `list_datasources` (type: `loki`) | ## Tool mapping | What to check | MCP tool | |---------------|----------| | Flux Kustomization / HelmRelease status | `flux: get_kubernetes_resources` | | Pod status, events, services | `flux: get_kubernetes_resources` | | Pod logs | `flux: get_kubernetes_logs` | | Pod CPU/memory usage | `flux: get_kubernetes_metrics` | | Force reconciliation | `flux: reconcile_flux_kustomization` / `reconcile_flux_helmrelease` | | Prometheus metrics | `grafana: query_prometheus` | | Application logs (Loki) | `grafana: query_loki_logs` | | Log error patterns | `grafana: find_error_pattern_logs` | | Firing alerts | `grafana: list_alert_rules` | | Dashboard visuals | `grafana: get_panel_image` | ## Workflow ### Step 1 — Identify the target Determine what the user wants to debug: - **Specific app** — e.g., "radarr is broken" → app=`radarr`, namespace=`media` - **Namespace-wide** — e.g., "media apps are down" → scan all apps in namespace - **Cluster-wide** — e.g., "what's broken" → scan all namespaces - **Specific symptom** — e.g., "reconciliation stuck" → focus on Flux resources Use the namespace mapping from the app skill: - `default` — homepage - `network` — adguard-home, cloudflare-dns, cloudflare-tunnel, envoy-gateway, k8s-gateway, unifi-dns - `monitoring` — alloy, gatus, grafana, kube-prometheus-stack, loki, ntfy, ntfy-alertmanager, smartctl-exporter, unpoller - `database` — cloudnativepg, redis - `media` — bazarr, flaresolverr, jellyfin, prowlarr, qbittorrent, radarr, recyclarr, seerr, sonarr - `security` — authentik - `storage` — volsync ### Step 2 — Flux GitOps layer Check Flux resources top-down. Run these in parallel when possible: 1. **Kustomization status** — check if the app's Kustomization is ready: ``` flux: get_kubernetes_resources apiVersion: kustomize.toolkit.fluxcd.io/v1 kind: Kustomization namespace: # the app's namespace (set by Kustomize transformer) name: # optional, omit for namespace-wide scan ``` 2. **HelmRelease status** — check if the HelmRelease is reconciled: ``` flux: get_kubernetes_resources apiVersion: helm.toolkit.fluxcd.io/v2 kind: HelmRelease namespace: name: ``` 3. **OCIRepository status** — check if the chart source is available: ``` flux: get_kubernetes_resources apiVersion: source.toolkit.fluxcd.io/v1 kind: OCIRepository namespace: name: ``` **What to look for:** - `Ready: False` with a message explaining why - `Suspended: true` — reconciliation paused - Dependency failures — a `dependsOn` target not ready - Source fetch errors — OCI registry unreachable or tag not found - Validation errors — invalid YAML or Helm values **If Flux resources look healthy**, the issue is at the Kubernetes or application layer — proceed to Step 3. **If Flux is stuck**, try reconciling: ``` flux: reconcile_flux_kustomization name: namespace: with_source: true ``` ### Step 3 — Kubernetes resource layer Check the workload resources. Run these in parallel: 1. **Pods** — are they running? ``` flux: get_kubernetes_resources apiVersion: v1 kind: Pod namespace: selector: app.kubernetes.io/name= ``` 2. **Events** — recent warnings or errors: ``` flux: get_kubernetes_resources apiVersion: v1 kind: Event namespace: ``` Filter events mentioning the app name in the results. 3. **PVCs** — storage bound? ``` flux: get_kubernetes_resources apiVersion: v1 kind: PersistentVolumeClaim namespace: name: ``` **What to look for:** - Pod status: `CrashLoopBackOff`, `ImagePullBackOff`, `Pending`, `OOMKilled` - Events: `FailedScheduling` (resource constraints), `FailedMount` (PVC issues), `Unhealthy` (probe failures) - PVC: `Pending` (no storage available) ### Step 4 — Application logs Fetch logs from the problematic pod. Use the Flux MCP for direct pod logs: ``` flux: get_kubernetes_logs pod_name: # from Step 3 pod listing container_name: app # most apps use "app" as container name pod_namespace: limit: 100 ``` If logs are also shipped to Loki, query for broader patterns: ``` grafana: query_loki_logs datasourceUid: logql: '{namespace="", app_kubernetes_io_name=""} |= "error" or |= "fatal"' limit: 50 ``` For pattern analysis over time: ``` grafana: find_error_pattern_logs name: " error investigation" labels: {"namespace": "", "app_kubernetes_io_name": ""} ``` ### Step 5 — Metrics and alerts (if relevant) Check resource utilization and alert state: 1. **Pod resource usage:** ``` flux: get_kubernetes_metrics pod_namespace: pod_selector: app.kubernetes.io/name= ``` 2. **Firing alerts:** ``` grafana: list_alert_rules ``` Filter for rules in `firing` or `pending` state. 3. **Custom PromQL** (when investigating specific symptoms): ``` grafana: query_prometheus datasourceUid: expr: startTime: now-1h queryType: instant ``` Useful queries: - Restart count: `kube_pod_container_status_restarts_total{namespace="", pod=~".*"}` - OOM kills: `kube_pod_container_status_last_terminated_reason{namespace="", reason="OOMKilled"}` - CPU throttling: `rate(container_cpu_cfs_throttled_seconds_total{namespace="", pod=~".*"}[5m])` - Memory usage: `container_memory_working_set_bytes{namespace="", pod=~".*", container!=""}` ### Step 6 — Summarize and recommend Present findings in this structure: ``` ## Diagnosis: **Status:** **Root cause:** **Evidence:** - - **Recommended fix:** 1. 2. ``` Common remediation patterns: - **Flux validation error** → fix the manifest in the repo, commit, and push - **Image pull failure** → check image tag/digest, registry availability - **CrashLoopBackOff** → check logs for startup errors, config issues, missing secrets - **OOMKilled** → increase memory limits in HelmRelease values - **PVC Pending** → check StorageClass, node capacity - **Probe failure** → check if port/path is correct, app startup time vs probe thresholds - **Dependency not ready** → debug the dependency first (recursive) - **SOPS decryption failure** → check age key is deployed, secret format is valid ## Important notes - Always discover datasource UIDs dynamically using `grafana: list_datasources` — never hardcode them. - Flux Kustomization namespace is the **app's namespace** (set by the Kustomize namespace transformer), not `flux-system`. - HelmRelease namespace follows the same pattern — it lives in the app's namespace. - When checking multiple apps, run tool calls in parallel to speed up diagnosis. - Do NOT take remediation actions (reconcile, delete, apply) without user confirmation — diagnosis only unless asked to fix. - Pod container names: most apps use `app`, but check the actual container name from the pod spec if `app` doesn't work. --- > Source: [MarkNygaard/home-ops](https://github.com/MarkNygaard/home-ops) — distributed by [TomeVault](https://tomevault.io).