nic-debugging

# Debugging and Troubleshooting ## Common Failure Modes ### NGINX Reload Failure **Symptom:** Controller logs show "reload failed" or NGINX returns error status. **Diagnosis:** 1. Check controller logs for the generated config that failed 2. Look for `nginx -t` output in logs — shows exact syntax error and line number 3. Common causes: - Unsanitized user string injected into config (missing `containsDangerousChars()` check) - Template guard missing (`{{- if }}` / `{{- with }}`) for optional field - Duplicate directive from conflicting policies - Invalid upstream when no endpoints available **Fix pattern:** - Find the template or config generation code that produced the bad directive - Add validation to reject the input earlier, OR fix the template guard - Verify with `make test` — snapshot tests catch most template output issues ### CRD Not Taking Effect **Symptom:** User applies VirtualServer/Policy but NGINX config doesn't change. **Diagnosis:** 1. Check CRD status: `kubectl get vs <name> -o yaml` — look at `.status.message` 2. Check controller logs for sync errors on that resource 3. Common causes: - Validation rejecting the resource (check `status.state: Invalid`) - Missing secret reference (TLS, JWT, OIDC secrets) - Policy referenced but not found in namespace - Resource conflicts (duplicate host/path) ### Controller Crash / Panic **Symptom:** Pod restarts, panic in logs. **Diagnosis:** 1. Check logs for the panic stack trace 2. Common causes: - Nil pointer on optional CRD field (forgot `*bool`/`*int` check) - Map access without nil check on `.Spec.X` field - Race condition in concurrent secret/config access 3. Look for the file:line in the stack trace → usually in `internal/configs/` or `internal/k8s/` ### Snapshot Test Failure **Symptom:** `make test` fails with snapshot mismatch. **Diagnosis:** 1. This means template output changed — could be intentional or regression 2. Review the diff shown in test output 3. If change is intentional: `make test-update-snaps`, then re-read `git diff -- '**/__snapshots__/**'` to confirm only the expected directives moved 4. If change is unintentional: your template edit had side effects — fix the template **The inverse failure is more dangerous:** you edited a `.tmpl` and `make test-update-snaps` produced **no** diff. That is not a pass — it means no fixture sets the field your new branch depends on. Add the fixture in `template_test.go` / `templates_test.go` and regenerate. ## Log Locations | Context | Location | What to look for | | --- | --- | --- | | Controller logs | Pod stdout/stderr | Sync errors, reload status, validation failures | | NGINX error log | `/var/log/nginx/error.log` in container | Config syntax errors, upstream failures | | NGINX access log | `/var/log/nginx/access.log` in container | Request routing verification | ## Validation and Diagnostic Tools | Tool | Command | Purpose | | --- | --- | --- | | Config test | `nginx -t` (inside container) | Validate NGINX config syntax | | CRD status | `kubectl get vs,vsr,ts,pol -A` | Check resource state | | Controller logs | `kubectl logs <pod> -n nginx-ingress` | Runtime errors | | Describe events | `kubectl describe vs <name>` | Kubernetes events for the resource | | Generated config | `kubectl exec <pod> -- cat /etc/nginx/conf.d/<file>` | Inspect actual generated NGINX config | ## Debugging Workflow 1. **Reproduce** — Get the exact error. Is it a reload failure? Wrong routing? Crash? 2. **Assess security impact** — Before diving into the fix, ask: - Is this bug exploitable? (Can external input trigger it?) - Does the failure expose sensitive data in logs or error messages? - Could an attacker craft input to reach this code path? - If exploitable: flag for security review BEFORE fixing 3. **Locate the layer** — Use logs and status to determine: - Validation layer? → `status.state: Invalid` with reason - Config generation? → Generated config has wrong directives - Template? → Snapshot test shows the issue - Controller? → Sync error in logs, resource not processed 4. **Isolate** — Find minimum CRD/annotation that triggers the issue 5. **Fix** — Make the change in the correct layer (don't fix templates for validation bugs) 6. **Verify** — `make test` passes, snapshot output is correct 7. **Prevent** — Add a test case that would catch this regression (include negative/malicious input tests if the bug was in a validation path) ## Config Generation Debugging When the generated NGINX config is wrong: 1. **Find the template struct** — Which struct feeds the template? Check `internal/configs/version2/http.go` (VS) or `internal/configs/version1/config.go` (Ingress) 2. **Find the config generator** — Where is the struct populated? Check `internal/configs/virtualserver.go` or `internal/configs/ingress.go` 3. **Find the template** — Which `.tmpl` file renders it? Check `internal/configs/version2/nginx-plus.virtualserver.tmpl` or the OSS variant 4. **Add a snapshot test** — Create a test case in the appropriate `_test.go` file with the input that triggers the bug, run `make test-update-snaps` to capture current (wrong) output, then fix and regenerate ## Common Gotchas When Debugging - NGINX config errors show line numbers in the GENERATED file, not your template — map back manually - Secret-related failures often show as "file not found" in NGINX logs (secret not written to filesystem yet) - Policy ordering matters — first matching policy wins, check `generatePolicies()` logic - Plus-only features will work in Plus template but silently produce invalid config in OSS template - `containsDangerousChars()` failures are validation errors and typically result in `status.state: Invalid` — check the CRD status message and controller logs ## Security-Sensitive Debugging When debugging an issue that involves user-provided input reaching NGINX config: 1. **Trace the input path** — From CRD field / annotation → validation → config struct → template → NGINX config file. Identify every point where sanitization SHOULD happen. 2. **Check for injection** — Can crafted input inject NGINX directives? Look for `;`, `{`, `}`, `$`, newlines, backticks in the user-controlled value. 3. **Verify the guard** — Does `containsDangerousChars()` or `ValidateEscapedString()` cover this path? If not, the bug is a security vulnerability. 4. **Never log secrets** — When debugging TLS/JWT/OIDC issues, mask credential values. Log key names and paths, not contents. 5. **Check RBAC** — If the issue involves unauthorized access, verify ServiceAccount permissions and RBAC role bindings before looking at code.