Gitops Audit

GitOps Audit Skill

Quick Start

GitOps audit verifies that your cluster state matches your git state by running three checks:

1. Cluster Drift: Find resources in cluster that aren't in git
2. Spec Drift: Find differences between git manifests and live cluster state
3. Code Violations: Find hardcoded Kubernetes configs in application code

Run the audit:

/gitops-audit

Expected flow:

1. Skill detects GitOps repository location
2. Runs three audit checks in parallel
3. Generates comprehensive report with severity levels (CRITICAL, WARNING, INFO)
4. Provides remediation steps

When to Load Additional References

The core workflow below is sufficient for most audits. Load these references for detailed patterns and troubleshooting:

For detailed drift patterns and examples:

Read `~/.claude/skills/gitops-audit/references/DRIFT-PATTERNS.md`

Use when: You need detailed examples of what each audit type detects, or want to understand specific drift patterns

For configuration and resource handling:

Read `~/.claude/skills/gitops-audit/references/KUBERNETES-RESOURCES.md`

Use when: Setting up configuration, handling CRDs/operators, or understanding exclusion patterns

For troubleshooting and edge cases:

Read `~/.claude/skills/gitops-audit/references/TROUBLESHOOTING.md`

Use when: Encountering issues, dealing with false positives, or handling special cases (HPA, operators, multi-cluster)

Core Workflow

Step 1: Detect GitOps Repository

Automatic detection methods:

• Check environment variable: GITOPS_REPO_PATH
• Search common locations: ~/gitops-repo, ~/projects/gitops, ~/k8s-manifests
• Parse git remotes: git remote -v | grep -E "(gitops|k8s|kubernetes|manifests)"

Ask user if not found:

GitOps repository not automatically detected.

Please provide the path to your GitOps repository:
- Local path: /path/to/gitops-repo
- Or I can clone it: git@github.com:org/gitops-repo.git

Detect repository structure:

• Helm: Check for charts/ directory
• Kustomize: Check for kustomization.yaml or base/ directory
• Plain YAML: Check for manifests/ or k8s/ directory

Step 2: Run Cluster Drift Check

Objective: Find resources in cluster that don't exist in git

Process:

# Get all resources in namespace
RESOURCES=$(kubectl api-resources --verbs=list --namespaced -o name | \
  xargs -n 1 kubectl get --show-kind --ignore-not-found -o name -n $NAMESPACE)

# For each resource
for RESOURCE in $RESOURCES; do
  KIND=$(echo $RESOURCE | cut -d'/' -f1)
  NAME=$(echo $RESOURCE | cut -d'/' -f2)

  # Search git for manifest
  FOUND=$(find $GITOPS_REPO -name "*.yaml" -o -name "*.yml" | \
    xargs grep -l "kind: $KIND" | \
    xargs grep -l "name: $NAME")

  if [ -z "$FOUND" ]; then
    echo "DRIFT: $RESOURCE not in git"
  fi
done

Report format:

🚨 CRITICAL: Resources in cluster NOT tracked in git

Namespace: production
- Deployment/debug-pod (created manually, age: 2 days)
  Action: Delete OR add manifest to gitops-repo/production/

- ConfigMap/temp-config (no manifest found, age: 5 hours)
  Action: Delete OR formalize in charts/app/config/

Step 3: Run Spec Drift Analysis

Objective: Find differences between git manifests and live cluster state

Process:

# Render manifest from git
if [ "$TYPE" = "helm" ]; then
  RENDERED=$(helm template $CHART -f $VALUES_FILE)
elif [ "$TYPE" = "kustomize" ]; then
  RENDERED=$(kustomize build $PATH)
else
  RENDERED=$(cat $MANIFEST)
fi

# Get live resource
LIVE=$(kubectl get $KIND $NAME -n $NAMESPACE -o yaml)

# Diff them
DIFF=$(kubectl diff -f <(echo "$RENDERED") 2>&1)

if [ -n "$DIFF" ]; then
  echo "SPEC DRIFT: $KIND/$NAME"
fi

Report format:

⚠️  WARNING: Spec drift detected (git ≠ cluster)

Deployment/api-service (production):
  Git: replicas=3
  Cluster: replicas=5 (DRIFT)
  Action: Reconcile via gitops-apply skill

ConfigMap/app-config (staging):
  Git: LOG_LEVEL=info
  Cluster: LOG_LEVEL=debug (DRIFT)
  Action: Update git OR revert cluster

Step 4: Run Code Violations Scan

Objective: Find hardcoded Kubernetes configs in application code

Process:

# Scan for inline YAML
grep -r -E "(apiVersion|kind:.*Deployment|kind:.*Service)" \
  --include="*.py" --include="*.go" --include="*.js" --include="*.ts" \
  --include="*.sh" --exclude-dir=tests --exclude-dir=node_modules

# Scan for kubectl commands
grep -r -E "kubectl\s+(apply|create|delete|patch|edit)" \
  --include="*.sh" --include="*.py" --include="*.js" --include="*.go" \
  --exclude-dir=tests

# Scan for hardcoded resource creation (Go)
grep -r -E "(CreateDeployment|CreateService|client\.Create\()" \
  --include="*.go" --exclude-dir=vendor

# Scan for hardcoded resource creation (Python)
grep -r -E "(create_namespaced_deployment|create_namespaced_service)" \
  --include="*.py"

Report format:

❌ CRITICAL: Hardcoded Kubernetes configs in application code

scripts/deploy.sh:45
  kubectl apply -f - <<EOF
    apiVersion: apps/v1
    kind: Deployment
    ...
  EOF

  Issue: Inline YAML in deployment script
  Action: Move to GitOps repo → charts/my-app/templates/deployment.yaml

Step 5: Generate Comprehensive Report

Report structure:

# GitOps Audit Report

**Status:** ❌ FAILED (3 CRITICAL, 5 WARNING issues)
**Date:** 2024-01-16 14:30:25 UTC
**Repository:** /home/user/gitops-repo
**Namespaces Audited:** production, staging

## Summary

| Check           | Status  | Critical | Warning | Info  |
| --------------- | ------- | -------- | ------- | ----- |
| Cluster drift   | ❌ FAIL | 2        | 0       | 0     |
| Spec drift      | ⚠️ WARN | 0        | 5       | 0     |
| Code violations | ❌ FAIL | 1        | 1       | 1     |
| **TOTAL**       | ❌ FAIL | **3**    | **6**   | **1** |

---

## 🚨 CRITICAL Issues (Must Fix)

[Detailed findings...]

## ⚠️ WARNING Issues (Should Fix)

[Detailed findings...]

## ℹ️ INFO Issues (Informational)

[Documented exceptions, test fixtures...]

---

## Remediation Steps

**Immediate Actions (CRITICAL - Do Today):**

- Delete or formalize debug-pod
- Refactor deploy.sh to use GitOps

**Recommended Actions (WARNING - This Week):**

- Reconcile all spec drift
- Document manual changes

**Prevention (Ongoing):**

- Enable kubectl mutation blocking hook
- Run weekly drift audits
- Monitor ArgoCD sync status

Configuration

Essential Configuration

# Namespaces to audit (auto-detected if not specified)
namespaces:
  - production
  - staging
  - dev

# GitOps repository (auto-detected if not specified)
gitops_repos:
  - path: /path/to/gitops-repo
    type: helm # helm | kustomize | plain-yaml

# System namespace exclusions (default)
exclude_namespaces:
  - kube-system
  - kube-public
  - kube-node-lease
  - argocd
  - flux-system

# Ephemeral resource exclusions (default)
exclude_resource_types:
  - Event
  - EndpointSlice
  - Lease
  - Pod
  - ReplicaSet

For detailed configuration options, read references/KUBERNETES-RESOURCES.md

Severity Classification

CRITICAL (Block Commit/PR)

Must be fixed immediately:

• Resources in cluster not tracked in git
• Hardcoded kubectl mutations in production code
• Inline Kubernetes YAML in application code (not tests)
• Secrets not encrypted in git

Why: Breaks GitOps single source of truth, prevents rollback, no audit trail

WARNING (Report But Don't Block)

Should be fixed soon:

• Spec drift (minor differences between git and cluster)
• Missing resource requests/limits
• ConfigMap/Secret content drift
• Hardcoded resources in operator code (verify it's an operator)

Why: Doesn't break GitOps entirely, may be temporary, has valid exceptions

INFO (Informational)

No action required:

• Resources in git not deployed to cluster (expected if feature-flagged)
• Test fixture resources
• kubectl commands in test code
• Properly documented exceptions

Why: Expected behavior, documented exceptions, no compliance risk

Integration with Other Skills

Automatic Invocation

Code Review Integration:

When code review detects Kubernetes-related changes, this skill runs automatically:

• Changes to *.yaml, *.yml in charts/, manifests/, k8s/, kubernetes/
• Changes to values*.yaml, Chart.yaml, kustomization.yaml
• Kubernetes client library imports (Go, Python, Node.js, TypeScript)

create-pr Skill Integration

Before creating PR:

• Run gitops-audit
• Block PR creation if CRITICAL issues found
• Include WARNING issues in PR description
• Proceed if clean

gitops-apply Skill Integration

After applying changes:

• Modify manifest → Commit → Push to git
• Wait for ArgoCD/Flux sync
• Run gitops-audit to verify cluster state matches git

Dependencies

Required Tools

• kubectl - Cluster access (REQUIRED)
• git - GitOps repository access (REQUIRED)

Optional Tools (Enhanced Functionality)

• helm - For Helm-based GitOps repos
• kustomize - For Kustomize-based repos
• yq or jq - YAML/JSON parsing
• argocd CLI - For ArgoCD integration
• flux CLI - For Flux integration

Fallback strategies:

• If helm not available: Parse Chart.yaml and values.yaml as plain YAML
• If yq/jq not available: Use grep/awk for basic field extraction
• If ArgoCD CLI not available: Use kubectl to check Applications

Best Practices

1. Run Regularly

• Production: Weekly (automated)
• Staging: Daily (automated)
• Development: On-demand
• Before Production Deploy: Always

2. Act on CRITICAL Issues Immediately

CRITICAL issues must be resolved before:

• Merging PRs
• Deploying to production
• Closing sprint/iteration

3. Document Exceptions

If allowing spec drift or hardcoded resources:

allowed_exceptions:
  - resource: "Deployment/feature-flag-service"
    namespace: "production"
    field: "replicas"
    reason: "Auto-scaled by HPA, drift expected"
    reviewed: "2024-01-15"

4. Monitor ArgoCD/Flux Sync Status

Complement audit with sync monitoring:

• Check ArgoCD dashboard regularly
• Set up Slack/email alerts for OutOfSync
• Review sync history for patterns

Related Skills

• gitops-apply: Apply changes via GitOps workflow (use instead of kubectl)
• helm-chart-writing: Create Helm charts for GitOps
• helm-argocd-gitops: Configure ArgoCD Applications
• safe-commit: Commit GitOps manifest changes safely
• create-pr: Create PRs for GitOps changes

Emergency Override

If user explicitly states "I acknowledge these drift findings":

1. Document the acknowledgment
2. Include findings in PR description
3. Create follow-up tickets for remediation
4. Set reminders for cleanup

This should be RARE - GitOps compliance is not optional.