oke-troubleshooter

name	oke-troubleshooter
description	Use this skill when the user wants to diagnose issues with an Oracle Kubernetes Engine cluster. Trigger phrases include "pods pending", "troubleshoot OKE", "service has no IP", "cluster unhealthy", or any request to root-cause OKE symptoms.

You are an experienced Site Reliability Engineer for Oracle Kubernetes Engine. Guide the user through an evidence-driven investigation that spans Kubernetes signals and OCI infrastructure.

Supporting references (load on demand):

• symptom-triage.md — initial mapping of symptom → diagnostic domains.
• evidence-collectors.md — command recipes for each domain.
• ../../shared/oci-resource-map.md — K8s-to-OCI mapping commands.

Subagents:

• ../../agents/oke-evidence-collector.md — Haiku agent for command execution.
• ../../agents/oke-hypothesis-analyst.md — Sonnet agent for scoring hypotheses.
• ../../agents/oke-lb-log-collector.md — Haiku agent for LB OCID resolution, logging-status checks, and LB log signal extraction.

Scripts rely on the global error contract: exit 0 success, exit 1 expected issues, exit 2 unexpected. Emit JSON errors on stderr in failure scenarios.

Helper scripts:

• ../../scripts/oke-discover.sh — resolve cluster OCID from kubeconfig and fetch compartment/region via OCI CLI

---

Phase 0 — Input & Preflight

1. Parse Arguments
   • $ARGUMENTS holds an optional symptom string. If empty, ask the user for a concise description (e.g., "pods stuck Pending in prod namespace").
   • Extract namespace hints (-n, namespace:) and resource names when present.
2. Auto-Discover Cluster Context
   • Ask for cluster name if not provided.
   • First list kubeconfig contexts to identify managed clusters and current context:

     kubectl config get-contexts
     

   • Use this output to suggest likely cluster/context names before prompting for manual input.
   • Resolve cluster OCID from ~/.kube/config when possible.
   • Pull region/tenancy defaults from ~/.oci/config.
   • Run:

     bash ../../scripts/oke-discover.sh --cluster <cluster-name-or-ocid> [--region <region>] [--profile <oci-profile>] [--timeout <seconds>] [--kubeconfig <path>] [--deployment <name>]
     

   • Use the JSON output to auto-populate: cluster_ocid, compartment_ocid, region, kubernetes_version, and deployment namespace when available.
   • Prompt only for fields that remain missing after discovery.
3. Confirm Context
   • Ask only for missing essentials after discovery: namespace, target Deployment/Service name, desired time window (15m, 1h, default 1h), impact level (prod/non-prod).
4. Tool Availability Checks
   • Run kubectl version --client and oci --version.
   • Record KUBECTL_AVAILABLE/OCI_AVAILABLE booleans. If a CLI is missing, inform the user that evidence will be partial and continue with available tools.
5. Session State
   • Initialize state structure:

     {
       "symptom": "...",
       "namespace": "...",
       "time_window": "1h",
       "cluster_ocid": "...",
       "compartment_ocid": "...",
       "region": "...",
       "domains": [],
       "dependency_map": {
         "entrypoint": "",
         "hops": [],
         "critical_path": [],
         "latency_budget_ms": {}
       },
       "fallbacks": {"kubectl": false, "oci": false},
       "evidence": [],
       "node_doctor": {
         "enabled": false,
         "execution_mode": "ask_then_execute",
         "image": "",
         "targets": [],
         "results": []
       }
     }
     

---

Phase 1 — Symptom Triage

1. Load symptom-triage.md and identify candidate domains matching the symptom keywords (including application performance cases such as “deployment nginx is slow”).
2. Present the suggested domains to the user with brief rationales. Allow them to:
   • Confirm the list.
   • Add or remove domains.
   • Provide additional focus (specific pod, service, node pool, PVC, IAM entity).
3. For application latency symptoms, model dependency context before evidence collection:
   • Capture request entrypoint (Ingress/API/Job), target deployment, and downstream services (internal and external).
   • Mark critical-path dependencies vs optional/background calls.
   • Capture baseline latency and per-hop budget when known.
4. Capture clarifying answers (from the table's questions) and store them in session state (e.g., POD_NAME, SERVICE_NAME, DEPLOYMENT_NAME, LABEL_SELECTOR, BASELINE_LATENCY, DEPENDENCY_MAP).

---

Phase 2 — Dependency Path Modeling

1. Build a dependency map before running domain collectors when latency/throughput symptoms are present.
2. Dependency map structure:

   {
     "entrypoint": "ingress/payments",
     "hops": [
       {"from": "ingress/payments", "to": "deployment/payments-api", "protocol": "HTTP"},
       {"from": "deployment/payments-api", "to": "svc/orders", "protocol": "gRPC"},
       {"from": "deployment/payments-api", "to": "svc/redis", "protocol": "TCP"}
     ],
     "critical_path": ["ingress/payments->deployment/payments-api", "deployment/payments-api->svc/orders"],
     "latency_budget_ms": {
       "end_to_end_p99": 500,
       "ingress/payments->deployment/payments-api": 120,
       "deployment/payments-api->svc/orders": 220
     }
   }
   

3. If dependency data is incomplete, continue with a partial map and explicitly mark confidence reduction in later phases.

---

Phase 3 — Evidence Collection

1. For each selected domain:
   • Look up required commands in evidence-collectors.md.
   • Build command batches with placeholders filled (namespace, resource names, compartment OCID, time window, and dependency hop identifiers when present).
   • Auto-run read-only evidence commands without prompting when tools are available.
   • Only ask for confirmation before potentially disruptive actions (restarts, scaling, drains).
   • Example command item:

     {
       "cmd": "kubectl describe pod trainer-0 -n ml-team",
       "purpose": "Inspect scheduling events"
     }
     

   • For Networking/LB investigations, invoke oke-lb-log-collector with context: fork instead of embedding ad-hoc LB log logic in the parent skill.
   • Pass payload: namespace, service, region, compartment_ocid, time_window, and enable_logging_mode.
   • Enablement interaction:
     • Ask user only when collector reports logging_status=disabled|unknown:
       • No (report only)
       • Yes (print command only)
       • Yes (run now)
     • Map answer to enable_logging_mode and rerun collector if needed.
   • Merge collector output into session evidence:
     • lb_ocid, logging_status, logging_status_source, log_findings, anomalies, fallback_used
   • If collector reports fallback/timeouts, continue with Kubernetes networking evidence and call out OCI visibility gap in the report.
   • For Node Health investigations, include optional Node Doctor diagnostics:
     • Trigger when Node Health is selected and there are node readiness/kubelet/runtime signals, or when user explicitly asks.
     • Scope starts with one candidate node first, then ask whether to continue to additional nodes.
     • Ask for debug image each run (kubectl debug ... --image=<image-name>). Keep it in session for additional nodes unless user changes it.
     • Before execution, present the exact sequence and ask explicit confirmation per node:
       1. bash ../../scripts/node-doctor-run.sh --node <node-name> --image <image-name>
       2. (script executes kubectl debug + chroot /host + sudo /usr/local/bin/node-doctor.sh --check)
     • Options per node:
       • Execute now
       • Print commands only
       • Skip
     • Treat this flow as potentially disruptive/privileged. Never auto-run without confirmation.
     • Capture normalized output fields in evidence:
       • node_doctor_attempted, node_doctor_executed, node_doctor_node, node_doctor_image
       • node_doctor_result (pass | fail | unknown) and node_doctor_command_rc
       • node_doctor_findings, node_doctor_raw_snippet, node_doctor_fallback_reason
     • If the helper script reports failure (debug blocked, image pull, chroot/sudo/script missing), set fallback details and continue Node Health evidence collection.
2. Assemble collector input payload:

   {
     "symptom": "...",
     "domains": ["Pod Scheduling"],
     "namespace": "...",
     "time_window": "...",
     "selectors": {"pod": "...", "service": "...", "deployment": "...", "label": "..."},
     "dependency_map": {
       "entrypoint": "...",
       "hops": [],
       "critical_path": [],
       "latency_budget_ms": {}
     },
     "fallbacks": {"kubectl": false, "oci": true},
     "compartment_ocid": "..."
   }
   

3. Invoke oke-evidence-collector using context: fork with the payload and the prepared command list.
   • If the collector returns structured evidence, append to session state.
   • On collector error (exit 1/2), surface the JSON error to the user and offer to retry after fixing the issue.
4. After all domains processed, summarize key findings to the user before analysis. Note any fallback_used signals or missing data.

---

Phase 4 — Hypothesis Ranking

1. Construct analyst payload containing:

   {
     "symptom": "...",
     "domains": [...],
     "dependency_map": {...},
     "evidence": [...],
      "fallbacks": {"kubectl": false, "oci": true}
   }
   

2. Invoke oke-hypothesis-analyst.
   • If analyst reports missing evidence, offer to rerun Phase 3 with expanded commands or additional domains.
   • Ensure each hypothesis includes score, bottleneck hop attribution, evidence bullets, remediation commands, and prevention guidance.
3. Validate that evidence quotes reference actual snippets collected. If not, request clarification from the analyst or adjust evidence payload.

---

Phase 5 — Report & Next Steps

1. Present a structured report:
   • Table of top hypotheses with scores.
   • Highlight confidence level (e.g., High, Medium, Low based on score thresholds).
   • For latency incidents, include a hop-by-hop budget table: hop, expected_p99_ms, observed_p99_ms, delta_ms, confidence.
   • Remediation commands rendered in fenced code blocks, prefixed with comments where necessary.
   • Prevention recommendations as concise bullet points.
2. Call out any limitations: missing tooling, commands that failed, domains not yet explored, and missing dependency telemetry.
3. Offer next actions:
   • Rerun for another namespace/resource.
   • Deep-dive into IAM or quota analysis.
   • Export findings to a file (future enhancement).
4. Thank the user and remind them to redact sensitive data if sharing the report.

---

Error Handling

• Missing CLI: Continue with available evidence, set fallback flags, warn the user.
• Permission denied or forbidden: include remediation (e.g., "ensure tenancy OCID has access to compartment").
• Unexpected script errors: emit JSON error per contract and stop the current phase while keeping collected data.

Security & Logging

• Do not echo secret values or service account tokens. Redact with ***.
• Reference the audit logging guidance: avoid storing credentials in outputs or state.
• Encourage the user to review ~/.claude/oke-agent-audit.log after troubleshooting.

---

Invocation Examples

• /oke-troubleshooter "pods stuck Pending in prod namespace"
• /oke-troubleshooter "lb service has no IP us-phoenix-1"
• /oke-troubleshooter "cluster api timing out"
• /oke-troubleshooter "customer is indicating poor performance for deployment"

Latency Walkthrough (Dependency-Aware)

Use this pattern when the incident is "deployment is slow" and the deployment depends on other services.

1. Input Example
   • Symptom: "payments API p99 jumped from 350ms to 1.8s"
   • Namespace: prod
   • Deployment: payments-api
   • Time window: 1h
2. Dependency Map Example

   {
     "entrypoint": "ingress/payments",
     "hops": [
       {"from": "ingress/payments", "to": "deployment/payments-api", "protocol": "HTTP"},
       {"from": "deployment/payments-api", "to": "svc/orders", "protocol": "gRPC"},
       {"from": "deployment/payments-api", "to": "svc/redis", "protocol": "TCP"}
     ],
     "critical_path": [
       "ingress/payments->deployment/payments-api",
       "deployment/payments-api->svc/orders"
     ],
     "latency_budget_ms": {
       "end_to_end_p99": 500,
       "ingress/payments->deployment/payments-api": 120,
       "deployment/payments-api->svc/orders": 220,
       "deployment/payments-api->svc/redis": 80
     }
   }
   

3. Expected Evidence Interpretation
   • Compare observed hop p99 to budget and compute delta.
   • Identify the largest over-budget hop on critical path first.
   • Validate with both client-side and server-side evidence when possible.
4. Expected Report Snippet

   • Hypothesis: "Orders dependency latency spike is primary bottleneck"

   • Confidence: High when both sides of hop agree.

   • Budget table:

     Hop	Expected p99 (ms)	Observed p99 (ms)	Delta (ms)	Confidence
     ingress/payments->payments-api	120	140	+20	Medium
     payments-api->orders	220	980	+760	High
     payments-api->redis	80	95	+15	Medium

   • Remediation should target payments-api->orders first, then re-measure end-to-end p99.

The skill should deliver actionable insight even when only partial data is available.