| name | service-mesh-debug |
| description | Diagnose and fix flaky e2e tests and general connectivity issues in service mesh environments (Kuma, Istio, Linkerd, Consul). Trigger when: a user mentions intermittent test failures, "test is flaky", e2e failures in CI that don't reproduce locally, Ginkgo/Gomega test files that fail sometimes, 503/connection refused errors, mTLS handshake failures, pods not getting traffic, xDS NACKs or warming resources, cert delivery timing issues, or "works locally but not in cluster". Covers Kuma flakiness patterns (timing races, xDS propagation delays, Envoy circuit breakers, mTLS readiness, Gomega misuse) AND universal mesh debugging (control plane connectivity, proxy lifecycle, certificate problems, traffic routing/policy, service discovery).
|
| allowed-tools | Read, Grep, Bash |
| user-invocable | true |
Fix Flaky E2E / Service Mesh Debugging Skill
Two modes: Flaky E2E Fix (Kuma/Ginkgo test files) and Mesh Connectivity Debug (live cluster issues).
Scope
- Designed for Kuma/Envoy e2e test flakiness and service mesh connectivity issues
- Not designed for unit test failures, application-level bugs, or non-mesh networking problems
- Assumes the test framework is Ginkgo/Gomega with the Kuma
test/framework/ helpers (Mode 1) or a live K8s cluster with mesh sidecar injection (Mode 2)
Mode 1: Flaky E2E Fix
Kuma e2e tests use Ginkgo/Gomega with a custom framework in test/framework/. Flakiness almost
always traces to one of ~11 known root causes. The fix is usually a 1-3 line change.
Process (Flaky E2E)
1. Read the test
Read the file the user provides (or grep for it). Focus on:
Eventually / Consistently calls and their timeout stringsExpect(...) calls inside Eventually blocks- Pod creation / deletion sequences
- Where traffic assertions happen relative to resource creation
- Any
time.Sleep calls
2. Diagnose: match to the taxonomy
Read references/root-causes.md in full before matching — it contains
code examples and fix patterns for all 11 root causes. The most common causes in order of frequency:
| # | Pattern | Fast signal |
|---|
| 1 | Short Eventually timeout | "30s" near gateway/policy/mTLS code |
| 2 | Missing xDS readiness gate | Traffic asserted immediately after policy apply |
| 3 | Bare Expect inside Eventually | Eventually(func() { Expect(...) }) — no g Gomega |
| 4 | Pod not available after create | WaitUntilNumPodsCreatedE without WaitUntilPodAvailableE |
| 5 | PodNameOfApp race after kill | Called immediately after KillAppPod |
| 6 | External component not awaited | SPIRE / cert-manager / Postgres creation without availability wait |
| 7 | xDS config diff before convergence | config_dump compare without Eventually wrapper |
| 8 | SDS secret timing | mTLS test fails with Secret is not supplied by SDS |
| 9 | Statistical tight margins | Traffic split % assertion with small N |
| 10 | Circuit breaker tripped | Concurrent test runs exhaust CB defaults |
| 11 | Outlier detection ejection | Fault-injection test leaves host ejected for 30s |
3. Apply the minimal fix
Read references/fix-patterns.md before applying any fix — it contains
copy-paste templates matched to each root cause.
Key rules:
- Change only what's flaky — keep the diff minimal so reviewers can verify the fix in isolation.
- Use the root-cause-specific fix pattern from fix-patterns.md. Never use
FlakeAttempts(n) as the primary fix because it hides root causes and lets the underlying race regress silently.
- Replace
time.Sleep with Eventually because sleep-based waits are fragile under variable CI load and slow down the suite unnecessarily.
- After fixing, run:
make format && make check
- If
make check reports failures, return to Step 2 with the error output and re-diagnose — the original root cause may have been misidentified.
4. Add AfterEachFailure if missing
If the test lacks a failure debug hook, suggest adding:
AfterEachFailure(func() {
DebugKube(KubeCluster, meshName, namespace)
})
This dumps CP logs, dataplane state, and pod info on failure — essential for future debugging.
5. Envoy diagnosis (if needed)
If the failure involves connectivity, xDS config, or mTLS and you need to guide the user through
live debugging, read references/envoy-debug.md before starting — it
contains the full admin API reference and 7-step diagnostic workflow.
For live cluster debugging, suggest running the scripts in scripts/ directly against the pod:
"${CLAUDE_SKILL_DIR}/scripts/envoy_snapshot.py" <pod> -n <namespace>
"${CLAUDE_SKILL_DIR}/scripts/xds_check.py" <pod> -n <namespace> --cluster <cluster-name>
"${CLAUDE_SKILL_DIR}/scripts/mtls_check.py" <pod> -n <namespace>
Scripts require only kubectl in PATH and Python 3.9+. No extra dependencies.
All scripts support --admin-port (default 9901) for non-Kuma meshes (Istio: 15000, Consul: 19000).
Mode 2: Mesh Connectivity Debugging
For general connectivity issues in any service mesh (503s, cert errors, traffic blocked, no healthy
hosts, policy denials), follow the 7-phase workflow.
Quick start
"${CLAUDE_SKILL_DIR}/scripts/mesh_health.py"
"${CLAUDE_SKILL_DIR}/scripts/xds_check.py" <pod> -n <ns>
"${CLAUDE_SKILL_DIR}/scripts/xds_check.py" <pod> -n <ns> --admin-port 15000 --container istio-proxy
"${CLAUDE_SKILL_DIR}/scripts/mtls_check.py" <pod> -n <ns>
"${CLAUDE_SKILL_DIR}/scripts/envoy_snapshot.py" <pod> -n <ns>
Classify the failure first
Read references/failure-taxonomy.md to classify the issue into
one of 6 categories before spending time on mesh-specific commands.
| Symptom | Category |
|---|
| All pods broken simultaneously | 1 – Control Plane |
| Single pod, no sidecar | 2 – Proxy Lifecycle |
Secret is not supplied by SDS | 3 – Certificates |
| TLS handshake failure | 3 – Certificates |
403 / UAEX flag | 4 – Policy |
UH / no healthy hosts | 5 – Service Discovery |
| Works on some nodes, not others | 6 – Infrastructure |
| Flaky in CI, passes locally | 1 or 3 |
Full debugging workflow
Read references/mesh-debug-workflow.md for the 7-phase
workflow with mesh-specific commands for Kuma, Istio, Linkerd, and Consul.
Framework helpers quick reference
| Helper | Location | Use for |
|---|
WaitForMesh | test/framework/resources.go | Multi-zone mesh sync |
WaitForResource | test/framework/resources.go | Any resource to appear |
DebugKube / DebugUniversal | test/framework/debug.go | State dump on failure |
AfterEachFailure | test/framework/ginkgo.go | Hook debug to failure only |
ControlPlaneAssertions | test/framework/debug.go | Assert CP not crashed |
CollectEchoResponse | test/framework/client/collect.go | HTTP connectivity check |
CollectFailure | test/framework/client/collect.go | Assert expected conn failure |
MustPassRepeatedly(n) | Gomega | Require N consecutive passes |
Within(timeout, task) | pkg/test/within.go | Goroutine with timeout |
Gomega timeout guidelines
| Context | Timeout | Rationale |
|---|
| Pod creation + readiness | "30s" | Scheduler + kubelet startup |
| Policy propagation (simple) | "30s" | CP reconcile + xDS push |
| Gateway / ingress policies | "60s" | Extra reconcile cycles |
| mTLS / SVID / cross-zone | "2m" | Cert issuance + KDS sync |
MustPassRepeatedly(5) | "2m" | Needs many attempts to confirm stable |
Polling interval: "1s" is standard. Use "500ms" only for fast local assertions.
Anti-patterns to flag
Expect(x) inside Eventually(func() { ... }) — must be Eventually(func(g Gomega) { g.Expect(x) })WaitUntilNumPodsCreatedE alone — always follow with WaitUntilPodAvailableE per pod- Asserting traffic before checking xDS config convergence
time.Sleep(N * time.Second) — replace with EventuallyFlakeAttempts(3) as first resort
Input: Test uses `Eventually(func() { Expect(resp.Instance).To(Equal("server")) }, "30s", "1s")`.
Diagnosis: Root cause #3 — bare Expect panics on first failure, bypassing retry logic.
Fix: `Eventually(func(g Gomega) { g.Expect(resp.Instance).To(Equal("server")) }, "30s", "1s").Should(Succeed())`
Input: `Install(policy)` immediately followed by traffic assertion; test passes locally but fails in CI.
Diagnosis: Root cause #2 — xDS propagation delay. CP reconcile + push takes 25s+ under CI load.
Fix: Insert xDS readiness gate (Pattern 2 in fix-patterns.md) between Install and traffic assertion.
Input: Test creates pods with `WaitUntilNumPodsCreatedE(t, opts, "app", 2, 30, 1*time.Second)` then immediately sends traffic.
Diagnosis: Root cause #4 — pods exist but containers are not ready. Created != Available.
Fix: Add `WaitUntilPodAvailableE(t, opts, podName, 30, 1*time.Second)` for each pod after the create wait.
