Nic Debugging

Debugging and Troubleshooting

Common Failure Modes

NGINX Reload Failure

Symptom: Controller logs show "reload failed" or NGINX returns error status.

Diagnosis:

1. Check controller logs for the generated config that failed
2. Look for nginx -t output in logs — shows exact syntax error and line number
3. Common causes:
   • Unsanitized user string injected into config (missing containsDangerousChars() check)
   • Template guard missing ({{- if }} / {{- with }}) for optional field
   • Duplicate directive from conflicting policies
   • Invalid upstream when no endpoints available

Fix pattern:

• Find the template or config generation code that produced the bad directive
• Add validation to reject the input earlier, OR fix the template guard
• Verify with make test — snapshot tests catch most template output issues

CRD Not Taking Effect

Symptom: User applies VirtualServer/Policy but NGINX config doesn't change.

Diagnosis:

1. Check CRD status: kubectl get vs <name> -o yaml — look at .status.message
2. Check controller logs for sync errors on that resource
3. Common causes:
   • Validation rejecting the resource (check status.state: Invalid)
   • Missing secret reference (TLS, JWT, OIDC secrets)
   • Policy referenced but not found in namespace
   • Resource conflicts (duplicate host/path)

Controller Crash / Panic

Symptom: Pod restarts, panic in logs.

Diagnosis:

1. Check logs for the panic stack trace
2. Common causes:
   • Nil pointer on optional CRD field (forgot *bool/*int check)
   • Map access without nil check on .Spec.X field
   • Race condition in concurrent secret/config access
3. Look for the file:line in the stack trace → usually in internal/configs/ or internal/k8s/

Snapshot Test Failure

Symptom: make test fails with snapshot mismatch.

Diagnosis:

1. This means template output changed — could be intentional or regression
2. Review the diff shown in test output
3. If change is intentional: make test-update-snaps, then re-read git diff -- '**/__snapshots__/**' to confirm only the expected directives moved
4. If change is unintentional: your template edit had side effects — fix the template

The inverse failure is more dangerous: you edited a .tmpl and make test-update-snaps produced no diff. That is not a pass — it means no fixture sets the field your new branch depends on. Add the fixture in template_test.go / templates_test.go and regenerate.

Log Locations

Validation and Diagnostic Tools

Debugging Workflow

1. Reproduce — Get the exact error. Is it a reload failure? Wrong routing? Crash?
2. Assess security impact — Before diving into the fix, ask:
   • Is this bug exploitable? (Can external input trigger it?)
   • Does the failure expose sensitive data in logs or error messages?
   • Could an attacker craft input to reach this code path?
   • If exploitable: flag for security review BEFORE fixing
3. Locate the layer — Use logs and status to determine:
   • Validation layer? → status.state: Invalid with reason
   • Config generation? → Generated config has wrong directives
   • Template? → Snapshot test shows the issue
   • Controller? → Sync error in logs, resource not processed
4. Isolate — Find minimum CRD/annotation that triggers the issue
5. Fix — Make the change in the correct layer (don't fix templates for validation bugs)
6. Verify — make test passes, snapshot output is correct
7. Prevent — Add a test case that would catch this regression (include negative/malicious input tests if the bug was in a validation path)

Config Generation Debugging

When the generated NGINX config is wrong:

1. Find the template struct — Which struct feeds the template? Check internal/configs/version2/http.go (VS) or internal/configs/version1/config.go (Ingress)
2. Find the config generator — Where is the struct populated? Check internal/configs/virtualserver.go or internal/configs/ingress.go
3. Find the template — Which .tmpl file renders it? Check internal/configs/version2/nginx-plus.virtualserver.tmpl or the OSS variant
4. Add a snapshot test — Create a test case in the appropriate _test.go file with the input that triggers the bug, run make test-update-snaps to capture current (wrong) output, then fix and regenerate

Common Gotchas When Debugging

• NGINX config errors show line numbers in the GENERATED file, not your template — map back manually
• Secret-related failures often show as "file not found" in NGINX logs (secret not written to filesystem yet)
• Policy ordering matters — first matching policy wins, check generatePolicies() logic
• Plus-only features will work in Plus template but silently produce invalid config in OSS template
• containsDangerousChars() failures are validation errors and typically result in status.state: Invalid — check the CRD status message and controller logs

Security-Sensitive Debugging

When debugging an issue that involves user-provided input reaching NGINX config:

1. Trace the input path — From CRD field / annotation → validation → config struct → template → NGINX config file. Identify every point where sanitization SHOULD happen.
2. Check for injection — Can crafted input inject NGINX directives? Look for ;, {, }, $, newlines, backticks in the user-controlled value.
3. Verify the guard — Does containsDangerousChars() or ValidateEscapedString() cover this path? If not, the bug is a security vulnerability.
4. Never log secrets — When debugging TLS/JWT/OIDC issues, mask credential values. Log key names and paths, not contents.
5. Check RBAC — If the issue involves unauthorized access, verify ServiceAccount permissions and RBAC role bindings before looking at code.