Kyverno Troubleshooting

Kyverno Troubleshooting

Diagnose failures in Kyverno — the Kubernetes-native policy engine that validates, mutates, generates, and cleans up resources using policies written as Kubernetes resources (no new language to learn).

Keywords

kyverno, policy, clusterpolicy, validate, mutate, generate, cleanup, admission, webhook, deny, block, enforce, audit, policyreport, clusterpolicyreport, rule, match, exclude, preconditions, background, policy-exception

When to Use This Skill

• Resource creation is blocked with "admission webhook denied the request"
• ClusterPolicy or Policy resources show errors or are not applied
• Mutations are not being applied to resources as expected
• Generate rules are not creating expected resources
• Policy reports show unexpected violations
• Webhook timeouts or failures are disrupting cluster operations
• Policies work in Audit mode but fail in Enforce mode
• Background scan results are missing or incomplete

When NOT to Use

• Resource creation fails but the error is not from an admission webhook → use k8s-namespace-troubleshooting
• Kyverno policies are delivered by Flux and not appearing → use flux-troubleshooting
• Pod Security Standards are enforced by the built-in PodSecurity admission (not Kyverno) → use k8s-security-hardening

Related Skills

• cert-manager-troubleshooting - Policies may affect certificate resources
• external-secrets-troubleshooting - Policies may block secret creation
• external-dns-troubleshooting - Policies may restrict annotations
• k8s-namespace-troubleshooting - General namespace diagnosis
• k8s-security-hardening - Security policies and Pod Security Standards
• k8s-platform-tenancy - Tenant isolation policies
• flux-troubleshooting - GitOps delivery of Kyverno policies

Quick Reference

Diagnostic Workflow

Resource blocked by Kyverno?
├─ Identify which policy blocked it
│   ├─ Error message contains policy name → Check that policy (Section 2)
│   └─ Generic webhook error → Check webhook health (Section 1)
├─ Policy exists but not enforcing?
│   ├─ Policy status Ready: False → Spec error (Section 2)
│   ├─ validationFailureAction: Audit → Only reports, doesn't block (Section 3)
│   └─ match/exclude misconfigured → Wrong resource selection (Section 3)
├─ Mutation not applying?
│   ├─ Mutate rule exists → Check match criteria and patch (Section 4)
│   └─ Order matters → Check policy ordering (Section 4)
├─ Generate rule not creating resources?
│   ├─ Trigger resource matched? → Check match/preconditions (Section 5)
│   └─ RBAC sufficient? → Kyverno SA needs create permission (Section 5)
└─ Webhook causing cluster issues?
    └─ Timeouts or failOpen config (Section 7)

Section 1: Controller Health

# Check all Kyverno components
kubectl get pods -n kyverno -o wide

# Kyverno v1.10+ has multiple controllers:
# - kyverno-admission-controller (validates/mutates on admission)
# - kyverno-background-controller (background scans, generate, cleanup)
# - kyverno-reports-controller (policy reports)
# - kyverno-cleanup-controller (cleanup policies)

# Controller logs
kubectl logs -n kyverno deploy/kyverno-admission-controller --tail=200
kubectl logs -n kyverno deploy/kyverno-background-controller --tail=200
kubectl logs -n kyverno deploy/kyverno-reports-controller --tail=100

# Check CRDs
kubectl get crd | grep kyverno

# Check webhook configurations
kubectl get validatingwebhookconfigurations | grep kyverno
kubectl get mutatingwebhookconfigurations | grep kyverno

# Check webhook endpoints
kubectl get endpoints -n kyverno

Component Issues

Section 2: Policy Status and Validation

# Check ClusterPolicy status
kubectl get clusterpolicy -o wide
kubectl describe clusterpolicy ${POLICY_NAME}

# Check Policy status (namespace-scoped)
kubectl get policy -n ${NS} -o wide
kubectl describe policy ${POLICY_NAME} -n ${NS}

# Check for policy spec errors
kubectl get clusterpolicy ${POLICY_NAME} -o json | jq '.status'

# List all rules in a policy
kubectl get clusterpolicy ${POLICY_NAME} -o json | jq '.spec.rules[].name'

Policy Validation Errors

Section 3: Validate Rules — Audit vs Enforce

# Check validation failure action
kubectl get clusterpolicy ${POLICY_NAME} -o jsonpath='{.spec.validationFailureAction}'

# Check per-rule overrides
kubectl get clusterpolicy ${POLICY_NAME} -o json | jq '.spec.rules[] | {name: .name, action: .validate.validationFailureAction}'

# Check policy reports for audit violations
kubectl get policyreport -n ${NS} -o json | jq '.items[].results[] | select(.result == "fail") | {policy: .policy, rule: .rule, resource: .resources[0].name, message: .message}'

# Check cluster-level violations
kubectl get clusterpolicyreport -o json | jq '.items[].results[] | select(.result == "fail") | {policy: .policy, rule: .rule, resource: .resources[0].name}'

Audit vs Enforce

Match and Exclude Patterns

# Common pattern: enforce on all namespaces except system ones
spec:
  rules:
    - name: check-labels
      match:
        any:
          - resources:
              kinds:
                - Pod
      exclude:
        any:
          - resources:
              namespaces:
                - kube-system
                - kyverno
                - cert-manager

Section 4: Mutate Rules

# Check if mutation is configured
kubectl get clusterpolicy ${POLICY_NAME} -o json | jq '.spec.rules[] | select(.mutate) | {name: .name, mutate: .mutate}'

# Check if mutation was applied (compare resource against expected)
kubectl get ${RESOURCE_TYPE} ${NAME} -n ${NS} -o yaml | grep -A5 "expected-field"

# Check mutation webhook
kubectl get mutatingwebhookconfigurations kyverno-resource-mutating-webhook-cfg -o yaml

# Logs for mutation errors
kubectl logs -n kyverno deploy/kyverno-admission-controller --tail=200 | grep -iE 'mutate|patch'

Mutation Types

Mutation Problems

Section 5: Generate Rules

# Check generate rules
kubectl get clusterpolicy ${POLICY_NAME} -o json | jq '.spec.rules[] | select(.generate) | {name: .name, kind: .generate.kind}'

# Check background controller (handles generate)
kubectl logs -n kyverno deploy/kyverno-background-controller --tail=200 | grep -iE 'generate|trigger'

# Check if generated resources exist
kubectl get ${GENERATED_KIND} -n ${TARGET_NS}

# Check RBAC for generated resource type
kubectl auth can-i create ${GENERATED_KIND} --as=system:serviceaccount:kyverno:kyverno-background-controller -n ${TARGET_NS}

Generate Rule Patterns

# Generate NetworkPolicy when namespace is created
spec:
  rules:
    - name: default-deny
      match:
        any:
          - resources:
              kinds:
                - Namespace
      generate:
        synchronize: true       # Keep in sync (update if policy changes)
        apiVersion: networking.k8s.io/v1
        kind: NetworkPolicy
        name: default-deny
        namespace: "{{request.object.metadata.name}}"
        data:
          spec:
            podSelector: {}
            policyTypes:
              - Ingress
              - Egress

Generate Problems

Section 6: Policy Exceptions

# List policy exceptions
kubectl get policyexception -A

# Check if an exception applies
kubectl get policyexception -A -o json | jq '.items[] | {name: .metadata.name, namespace: .metadata.namespace, policies: [.spec.exceptions[].policyName], match: .spec.match}'

Section 7: Webhook Issues

# Check webhook configuration
kubectl get validatingwebhookconfigurations kyverno-resource-validating-webhook-cfg -o yaml
kubectl get mutatingwebhookconfigurations kyverno-resource-mutating-webhook-cfg -o yaml

# Check webhook failure policy
kubectl get validatingwebhookconfigurations kyverno-resource-validating-webhook-cfg -o jsonpath='{.webhooks[*].failurePolicy}'

# Check webhook timeouts
kubectl get validatingwebhookconfigurations kyverno-resource-validating-webhook-cfg -o jsonpath='{.webhooks[*].timeoutSeconds}'

# Check namespace exclusions
kubectl get validatingwebhookconfigurations kyverno-resource-validating-webhook-cfg -o json | jq '.webhooks[].namespaceSelector'

Webhook Failure Modes

Webhook Problems

Emergency: Kyverno Blocking All Cluster Operations

If Kyverno is blocking all cluster operations and cannot recover, recommend the following to the user:

WARNING: Both options disable all Kyverno policy enforcement. Recommend these only when the cluster is operationally blocked.

Policy Testing

# Dry-run a resource against admission policies
kubectl apply -f resource.yaml --dry-run=server

# Use Kyverno CLI for offline testing
kyverno apply policy.yaml --resource resource.yaml

# Test all policies against a resource
kyverno apply /path/to/policies/ --resource resource.yaml

# Generate policy report for existing resources
kyverno apply policy.yaml --cluster

MCP Tools Available

When the appropriate MCP servers are connected, prefer these over raw kubectl where available:

• mcp__flux-operator-mcp__get_kubernetes_resources - Query ClusterPolicies, Policies, PolicyReports, webhooks
• mcp__flux-operator-mcp__get_kubernetes_logs - Retrieve Kyverno admission and background controller logs
• mcp__flux-operator-mcp__get_kubernetes_metrics - Check Kyverno resource consumption

Common Mistakes

Behavioural Guidelines

1. Identify the policy first — The admission error message contains the policy and rule name. Start there.
2. Check Audit vs Enforce — A policy in Audit mode allows resources through. Verify the enforcement mode before deeper debugging.
3. Exclude system namespaces — Always verify that kube-system, kyverno, and critical infrastructure namespaces are excluded.
4. Test with dry-run — Use kubectl apply --dry-run=server to test without persisting.
5. Check webhook health before policy logic — If the webhook is down with Fail policy, everything is blocked regardless of policy content.
6. Never weaken policies without understanding why — Recommend switching to Ignore failurePolicy or deleting webhooks only as an emergency measure, not a fix.
7. Check mutation before validation — Mutations are applied first. A missing mutation can cause a subsequent validation failure.
8. Review PolicyReports — For Audit mode policies, PolicyReports contain the violation details.
9. Report what you find, recommend what to change — This skill is diagnostic-only. Present findings and recommend commands for the user to execute.