triage-logs-with-stern

# Triage Logs with Stern Skill **Goal:** Tail logs across many pool pods at once, strip the variable parts of each error line, and rank by frequency to find the one recurring pattern that explains the symptom — instead of reading pod-by-pod. > This complements `diagnose_with_observer`'s `references/log-tracing.md`, which traces a *known* failure through a single call chain (gateway → multiorch → pooler → postgres). Use this skill instead when you don't yet know which component or pod is at fault and need to scan broadly first. ## 1. Ensure Stern is Available ```bash which stern || brew install stern ``` ## 2. Tail Broadly, Filter by Level Stern's first positional arg is a pod-name regex (not a label selector), so it matches across shards/cells without needing exact pod names: ```bash KUBECONFIG=kubeconfig.yaml stern 'mgc-.*-postgres-.*-inf-p' -i "error" ``` To scope to a single cluster or component instead, use `-l` with the same labels as the rest of the toolchain: ```bash KUBECONFIG=kubeconfig.yaml stern -l app.kubernetes.io/component=shard-pool -i "error" ``` Useful flags: | Flag | Effect | |------|--------| | `-i "<pattern>"` | Case-insensitive include filter on log line | | `-e "<pattern>"` | Exclude filter (strip known-noisy lines) | | `-c postgres` / `-c multipooler` | Restrict to one container in multi-container pods | | `--since 1h` | Limit to a time window instead of streaming live | | `--no-follow` | One-shot dump instead of tailing (best for triage) | | `-o extended` | Include pod/container/timestamp on every line | For triage, prefer a bounded, non-following dump over live tailing: ```bash KUBECONFIG=kubeconfig.yaml stern 'mgc-.*-postgres-.*-inf-p' -i "error" --since 1h --no-follow -o extended > /tmp/pooler-errors.log ``` ## 3. Rank Recurring Patterns Raw error lines differ by pod name, timestamp, and IDs, which hides the fact that they're the same root cause. Strip the variable tokens, then count: ```bash grep -oE '"message":"[^"]+"' /tmp/pooler-errors.log \ | sed -E 's/[0-9a-f]{8,}|schema "[^"]+"/<VAR>/g' \ | sort | uniq -c | sort -rn | head -20 ``` The message with the highest count is almost always the actual root cause — one pattern hitting every pod outranks incidental one-off errors. This is exactly how `schema "pg_pgrst_no_exposed_schemas" does not exist` was found: it was the only message that recurred across every pooler, once schema names were normalized to `<VAR>`. ## 4. Trace the Pattern to Code Once you have the dominant message, grep for the literal string (or the closest static substring around the variable part) in the operator and upstream multigres source to find where it's emitted and why: ```bash grep -rn "does not exist" --include="*.go" . ``` Cross-reference with `references/known-patterns.md` for signatures already triaged before, and with `diagnose_with_observer/references/code-investigation.md` to decide whether the fix belongs in the operator or upstream. ## 5. Report State the dominant pattern, its occurrence count relative to total error lines, the pods/components it spans, and the code location — not a list of every distinct error seen.