argo-cluster-debug

# Argo Cluster Debug Debug and troubleshoot the full Argo ecosystem on live Kubernetes clusters. ## Prerequisites | Tool | Required | Purpose | |------|----------|---------| | kubectl | Yes | CRD inspection, pod logs, resource status | | argocd | No | Richer Application output, diff, sync operations | | argo | No | Workflow logs, node inspection | | kubectl-argo-rollouts | No | Rollout status, step details, promotion | ## How This Skill Works The debug skill provides **systematic workflows** — each is a sequence of checks that traces the problem from surface symptoms to root cause. The value is in the completeness of the investigation, not just finding the first error. **Before starting any workflow:** 1. Check which CLIs are available (argocd, argo, kubectl-argo-rollouts, kubectl) 2. Confirm the current cluster context 3. Load `references/troubleshooting.md` — it maps symptoms to causes and saves time **General rules:** - Prefer specialized CLIs when available (richer output), fall back to `kubectl get -o yaml` - Read-only by default — never sync, promote, or delete unless explicitly asked - Analyze `.status.conditions` as the primary diagnostic signal for every Argo CRD - Complete the full workflow — don't stop after finding one issue, cascading problems are common ## Cluster Context - If the user specifies a cluster context, switch to it: ```bash kubectl config use-context <context-name> ``` If using argocd CLI with a remote server: ```bash argocd context <context-name> ``` - If no cluster is specified, use the current context. Confirm it: ```bash kubectl config current-context ``` - Always verify Argo CD installation before Application/ApplicationSet debugging. Check for the argocd namespace and CRDs. ## Debugging Workflows Execute the workflow that matches the user's problem. Each workflow is self-contained. Work through the steps in order — each step builds context for the next. --- ### Workflow 1: Argo CD Installation Check Use when: user asks if Argo CD is installed, reports general Argo CD failures, or as a prerequisite before Application debugging. **Steps:** 1. Check for Argo CD CRDs: ```bash kubectl get crd | grep argoproj.io ``` Expected CRDs: `applications.argoproj.io`, `appprojects.argoproj.io`, `applicationsets.argoproj.io`. If missing, Argo CD is not installed. 2. Identify the Argo CD namespace. It defaults to `argocd` but can be customized: ```bash kubectl get pods --all-namespaces -l app.kubernetes.io/part-of=argocd --no-headers | awk '{print $1}' | sort -u ``` 3. Check all pods in the Argo CD namespace: ```bash kubectl get pods -n argocd -o wide ``` 4. Verify core components are running. All required components must be `Running` with `READY` containers: | Component | Label Selector | Required | |-----------|---------------|----------| | argocd-server | `app.kubernetes.io/name=argocd-server` | Yes | | argocd-repo-server | `app.kubernetes.io/name=argocd-repo-server` | Yes | | argocd-application-controller | `app.kubernetes.io/name=argocd-application-controller` | Yes | | argocd-applicationset-controller | `app.kubernetes.io/name=argocd-applicationset-controller` | Yes | | argocd-redis | `app.kubernetes.io/name=argocd-redis` | Yes | | argocd-dex-server | `app.kubernetes.io/name=argocd-dex-server` | Yes | | argocd-notifications-controller | `app.kubernetes.io/name=argocd-notifications-controller` | No (optional) | **Note:** `argocd-notifications-controller` is not deployed in all installations. Its absence should not be treated as a failure. 5. If argocd CLI is available, check version compatibility: ```bash argocd version ``` Compare client and server versions. Mismatches beyond minor version can cause issues. 6. Check the argocd-cm ConfigMap for server configuration: ```bash kubectl get configmap argocd-cm -n argocd -o yaml ``` Look for: `url` (server URL), `repositories` (legacy repo config), `resource.customizations` (custom health checks), `kustomize.buildOptions`, `configManagementPlugins`. 7. Check argocd-rbac-cm for RBAC configuration: ```bash kubectl get configmap argocd-rbac-cm -n argocd -o yaml ``` Look for: `policy.default`, `policy.csv`, `scopes`. 8. If any component is CrashLoopBackOff or not ready, get its logs: ```bash kubectl logs -n argocd -l app.kubernetes.io/name=<component-name> --tail=100 --previous kubectl logs -n argocd -l app.kubernetes.io/name=<component-name> --tail=100 ``` 9. Check Argo CD events: ```bash kubectl get events -n argocd --sort-by='.lastTimestamp' | tail -30 ``` --- ### Workflow 2: Application Debugging Use when: user reports Application sync failure, health degradation, OutOfSync, or unknown status. **Steps:** 1. Get the Application resource: - argocd CLI: ```bash argocd app get <name> -o yaml ``` - kubectl: ```bash kubectl get application <name> -n argocd -o yaml ``` If the Application is in a non-default namespace, the user must specify it or check: ```bash kubectl get applications --all-namespaces | grep <name> ``` 2. Extract and assess sync status from `.status.sync.status`: | Status | Meaning | |--------|---------| | Synced | Live state matches desired state | | OutOfSync | Live state differs from desired state | | Unknown | Comparison could not be performed | 3. Extract and assess health status from `.status.health.status`: | Status | Meaning | |--------|---------| | Healthy | All resources healthy | | Degraded | One or more resources failed | | Progressing | Resources are being rolled out | | Suspended | Resource is paused (e.g., suspended CronJob, paused Rollout) | | Missing | Resource does not exist in the cluster | | Unknown | Health assessment not available | 4. Check `.status.conditions` for error messages. Common conditions: - `ComparisonError` — manifest generation failed - `SyncError` — sync operation failed - `InvalidSpecError` — Application spec is invalid - `OrphanedResourceWarning` — resources exist outside the Application's management 5. Check `.status.operationState` for the last sync operation: - `.status.operationState.phase` — Succeeded, Failed, Error, Running - `.status.operationState.message` — error detail - `.status.operationState.syncResult.resources` — per-resource sync results 6. If OutOfSync, analyze the diff: - argocd CLI: ```bash argocd app diff <name> ``` - kubectl: inspect `.status.resources` and compare `status` field per resource. Look for resources with `status: OutOfSync`. 7. Check the source configuration. Applications may use either `spec.source` (single-source) or `spec.sources[]` (multi-source, GA since v2.6). Check both: ```bash # Check if multi-source kubectl get application <name> -n argocd -o jsonpath='{.spec.sources}' 2>/dev/null | grep -q '\[' && echo "multi-source" || echo "single-source" # Single-source kubectl get application <name> -n argocd -o jsonpath='{.spec.source}' | python3 -m json.tool # Multi-source — iterate all sources kubectl get application <name> -n argocd -o jsonpath='{range .spec.sources[*]}repoURL={.repoURL} targetRevision={.targetRevision} path={.path} chart={.chart}{"\n"}{end}' ``` For multi-source Applications, each source can fail independently — verify `repoURL`, `targetRevision`, `path`, and Helm `valueFiles` for every entry in `spec.sources[]`. 8. If repo-server is failing to render manifests, check repo-server logs: ```bash kubectl logs -n argocd -l app.kubernetes.io/name=argocd-repo-server --tail=100 ``` 9. Inspect managed resources. List resources from `.status.resources` with non-Healthy or non-Synced status: ```bash kubectl get application <name> -n argocd -o jsonpath='{range .status.resources[*]}{.kind}/{.namespace}/{.name}: sync={.status}, health={.health.status}{"\n"}{end}' ``` 10. For each unhealthy managed resource, inspect it: ```bash kubectl get <kind> <name> -n <namespace> -o yaml kubectl describe <kind> <name> -n <namespace> kubectl get events -n <namespace> --field-selector involvedObject.name=<name> ``` 11. If managed resource is a Deployment/StatefulSet with failing pods: ```bash kubectl get pods -n <namespace> -l <label-selector> --sort-by='.status.startTime' kubectl logs -n <namespace> <pod-name> --tail=100 kubectl logs -n <namespace> <pod-name> --previous --tail=100 ``` 12. Produce a root cause analysis report (see Report Format). --- ### Workflow 3: ApplicationSet Debugging Use when: user reports ApplicationSet not generating Applications, generating wrong Applications, or ApplicationSet errors. **Steps:** 1. Get the ApplicationSet: ```bash kubectl get applicationset <name> -n argocd -o yaml ``` 2. Check `.status.conditions` for errors. Key conditions: - `ErrorOccurred` — generator or template rendering failed - `ParametersGenerated` — generator output status - `ResourcesUpToDate` — template application status 3. Identify the generator type and check its configuration: | Generator | Common Issues | |-----------|--------------| | list | Empty `elements` array | | git (directories) | `repoURL` inaccessible, `directories` path pattern wrong | | git (files) | File format invalid, path pattern wrong | | cluster | No matching clusters, label selector wrong | | pull request | API credentials missing/expired, no open PRs matching filter | | scmProvider | API credentials missing, org/owner wrong | | matrix/merge | Inner generator errors, key conflicts | | plugin | Plugin ConfigMap missing, generator RPC error | 4. List generated Applications: ```bash kubectl get applications -n argocd -l 'app.kubernetes.io/managed-by=applicationset-controller' --show-labels ``` Cross-reference with expected output from the generator. 5. If no Applications generated, check the applicationset-controller logs: ```bash kubectl logs -n argocd -l app.kubernetes.io/name=argocd-applicationset-controller --tail=200 ``` 6. If some Applications are generated but failing, debug each one using Workflow 2. 7. For progressive sync issues, check `spec.strategy`: ```bash kubectl get applicationset <name> -n argocd -o jsonpath='{.spec.strategy}' | python3 -m json.tool ``` Check `rollingSync.steps`, `maxUpdate` values, and whether unhealthy apps are blocking progression. --- ### Workflow 4: Rollout Debugging Use when: user reports canary/blue-green rollout stuck, aborted, or degraded. **Steps:** 1. Get the Rollout status: - kubectl-argo-rollouts: ```bash kubectl argo rollouts get rollout <name> -n <namespace> ``` - kubectl: ```bash kubectl get rollout <name> -n <namespace> -o yaml ``` 2. Check the phase from `.status.phase`: | Phase | Meaning | |-------|---------| | Healthy | Rollout completed successfully | | Paused | Waiting for manual promotion or analysis | | Progressing | Actively rolling out | | Degraded | Rollout encountered errors | | Aborting | Rollback in progress | | Aborted | Rollback completed | 3. For canary rollouts, check current step and step index: ```bash kubectl get rollout <name> -n <namespace> -o jsonpath='{.status.currentStepIndex}' kubectl get rollout <name> -n <namespace> -o jsonpath='{.spec.strategy.canary.steps}' ``` Identify which step the rollout is paused/stuck at. 4. If paused, determine the reason: - `pause` step: manual approval required. Check `.status.pauseConditions`. - `analysis` step: AnalysisRun in progress. Proceed to step 5. - `setWeight` step: traffic shift issue. Proceed to step 8. 5. Get AnalysisRuns for this Rollout: ```bash kubectl get analysisrun -n <namespace> --sort-by='.metadata.creationTimestamp' ``` Filter by Rollout ownership: ```bash kubectl get analysisrun -n <namespace> -o yaml | grep -A5 'ownerReferences' | grep '<rollout-name>' ``` Or use the revision label: ```bash kubectl get analysisrun -n <namespace> -l rollouts-pod-template-hash=<hash> ``` 6. Inspect the AnalysisRun: ```bash kubectl get analysisrun <analysisrun-name> -n <namespace> -o yaml ``` Check: - `.status.phase` — Running, Successful, Failed, Error, Inconclusive - `.status.metricResults[*].phase` — per-metric status - `.status.metricResults[*].measurements` — individual measurement values - `.status.metricResults[*].message` — error messages from the provider 7. If analysis failed, check the AnalysisTemplate: ```bash kubectl get analysistemplate <template-name> -n <namespace> -o yaml ``` Verify: - `successCondition` / `failureCondition` expressions - `provider` configuration (Prometheus URL, Datadog API key, web query, etc.) - `interval`, `count`, `failureLimit`, `inconclusiveLimit` 8. Check traffic management configuration: - Istio: ```bash kubectl get virtualservice -n <namespace> -o yaml kubectl get destinationrule -n <namespace> -o yaml ``` - NGINX Ingress: ```bash kubectl get ingress -n <namespace> -o yaml ``` Check `canary-*` annotations. - ALB: ```bash kubectl get ingress -n <namespace> -o yaml ``` Check `alb.ingress.kubernetes.io/*` annotations. - Traefik: ```bash kubectl get traefikservice -n <namespace> -o yaml ``` 9. Check ReplicaSets — compare stable vs canary: ```bash