debug

name	debug
description	Debug methodology — triage, parallel investigation, and synthesis.
emit	both

Debug Methodology

Triage First

Classify the bug before diving in:

• Crash / panic → check logs, attach a debugger.
• Wrong behavior → trace the code path, form hypotheses, then verify with a targeted test.
• Only in multi-service / multi-process flows → reproduce with an end-to-end test against the full stack.
• Flaky test → run in a loop with stress / race detection enabled.
• Works locally, fails in CI → diff the environment (versions, env vars, filesystem, time, network) before assuming a code bug.

Parallel Investigation

If your environment supports parallel agents, split the work into three independent tracks:

• Researcher — trace code paths, form hypotheses, check git log for recent changes that touch the affected area.
• Tester — write a failing test that isolates the bug. Top-down bisection: start at the outermost reproducing layer (e2e), then narrow to the unit.
• Reproducer — add diagnostic logging, exercise the path in a running system, collect runtime evidence (request IDs, error chains, timing).

Working solo? Run the three passes sequentially: research → reproduce → isolate.

Synthesis

Combine findings from all tracks before proposing a fix:

• Root cause with confidence level (High / Medium / Low).
• Evidence from each investigation track — which path, which test, which log line.
• Recommended fix approach — hand off to an implementer; don't fix in debug mode.

Discipline

• Root cause over symptom. Trace the failing call through the real code path and read the actual server / pod / service logs — don't pattern-match the error message and guess. The first plausible explanation is often a symptom one layer down from the cause.
• Reproduce before you guess. A bug you can't trigger on demand is one you can't verify a fix for.
• One hypothesis per test. A failing test that "exercises the bug area" doesn't prove the hypothesis; one that fails for exactly one reason does.
• Check connectivity and credentials first. For anything that crosses a process or cluster boundary, the top failure mode is stale connection state, not a logic bug: can the caller actually reach the target API/endpoint? Is the kubeconfig context current and pointed at the right cluster? Are the credentials live? Confirm the wire before you debug the payload.
• Fail fast — isolate and name the one blocker. When stuck, don't loop re-running the same failing command with small tweaks. Identify the single specific thing that's blocking (a 403, an unreachable host, an undefined symbol) and state it plainly. A precise blocker is actionable; "it's broken" is not.
• Don't widen the diff. Touch only what the evidence indicts; refactoring "while you're in there" turns a 1-line fix into a 200-line PR that needs its own review.

Forge-Specific Triage

On top of the generic triage above, common forge-shaped bug classes:

• Stale generated code / config drift → if symbols are "undefined" or a build breaks right after a proto / forge.yaml / schema change, suspect stale gen before you chase the code. Run forge generate, then forge audit to surface drift, then retest. Forge generates from proto + forge.yaml; stale gen masquerades as a bug in hand-written code.
• Can't reach a service / cross-cluster flow fails → verify connectivity and credentials before assuming a logic bug. Each service deploys to its own K8sCluster context (multi-cluster is native); a "service unreachable" or auth failure is usually a stale or wrong kubectl context, an expired credential, or a service that isn't up — not a code path. Confirm the context is current and the target responds first.
• Broken DI wiring → check the explicit composition (internal/app/compose.go NewComponents, off the owned internal/app/providers.go OpenInfra). Deps are interface-typed fields resolved by type off Infra, so a missing required provider is a compile error (NewComponents won't build) or a forge generate "no provider" report, not a silent nil — but a wrong fill (an unintended optional dep left nil) can still surface as a nil-pointer panic deep in handler code. There is no generated wire_gen.go/bootstrap.go to inspect; the wiring is NewComponents filling typed Deps off Infra.
• Mock vs real divergence → tests pass with generated mocks but the real adapter fails. Re-run the integration suite (forge test integration) before chasing a unit-test ghost.
• Proto-DB drift → entity types and DB schema evolve independently; forge audit flags the mismatch. If the symptom is "column X not found" in a handler that names the right struct field, audit first.

Forge Debug Tools

forge introspect handlers      # every RPC path the binary registers — localize the fault
forge api curl <svc.method>    # build a copy-pasteable Connect curl from the shell
forge cluster logs --service X # kubectl-backed log tail for one service (single cluster)
forge debug start <svc>        # build + attach Delve debugger
forge debug start --attach PID # attach Delve to a live process (don't `stop` it — see below)
forge debug break/continue/eval# breakpoints, resume, evaluate in the debug session
forge debug stop               # ends the session AND kills the debugged process
forge test --service <name> -V # verbose isolated test runs
forge test e2e                 # full-stack reproduction
forge test --race              # run tests with race detector
forge generate                 # regenerate code (use when stale gen is suspected)

Pick the tool for the job — and know where each stops short:

• forge introspect handlers to localize. Prints every RPC path the assembled binary serves. If the failing RPC isn't there, the fault is a downstream/remote hop, not this binary — that one check collapses the search space.
• forge api curl <service.method> to exercise an endpoint. Builds and runs a Connect request from the shell, but it stops at the auth interceptor — it does not mint a token. Use it for shape/connectivity; supply your own credential for an authed call.
• forge cluster logs --service <name> for the symptom. kubectl-backed, no file-grepping — but it tails only the owner cluster (the one cluster name in config). For a multi-cluster app, set the context and run kubectl --context <other> -n <ns> logs <pod> for the other half.
• forge debug start <svc> (Delve) — two sharp edges. In a multi-binary repo start <service> can mis-build (it falls back to ./cmd/...); pass an explicit path/service so it builds the binary you mean. And forge debug stop after --attach kills the live process — detach instead of stop when you've attached to something you need to keep running.

Smoke/doctor are NOT app-flow proofs. A green forge smoke / forge doctor checks listeners, local compose, and telemetry — never app-flow invariants. They (and forge cluster status, green whenever pods are Running) can all be green while the actual flow is broken. The only things that prove an app-flow fix:

1. a declarative, exit-coded app-health assertion (model: a project doctor:<flow> task that fails non-zero when the invariant is violated), and
2. a full forge test e2e.

Generic forge tools localize and surface evidence; they do not certify the fix.

Use chrome-devtools MCP tools for frontend bugs (snapshots, console, network).

Sub-Skills (forge)

The three investigation tracks above each have a dedicated forge sub-skill with concrete patterns:

• reproduce — runtime evidence collection and e2e reproduction.
• isolate — top-down bisection from e2e to unit test.
• investigate — hypothesis formation and code tracing.
• multi-cluster-e2e — a workload stuck Pending / not-ready, or a flow that fails only in a k3d multi-cluster e2e env: read the pod status before the network, hold the pod up on failure, and the nested-cluster image-pull / host-gateway-DNS / path-MTU gotchas.

Observability (Grafana LGTM)

docker-compose runs a Grafana LGTM stack with traces, metrics, logs, and continuous profiling.

• Grafana UI: http://localhost:3000 (no login needed — anonymous admin)
• Traces: Grafana → Explore → Tempo. Find slow requests, trace cross-service calls.
• Metrics: Grafana → Explore → Prometheus. Query http_server_request_duration_seconds etc.
• Logs: Grafana → Explore → Loki. Search structured logs.
• Profiles: Grafana → Explore → Pyroscope. CPU, heap, goroutine, mutex profiles from the app's pprof endpoint.

The app auto-connects: OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 pushes traces and metrics. PPROF_ADDR=localhost:6060 exposes pprof for Pyroscope scraping.

For LLM-driven observability, enable the Grafana MCP server from .mcp.json.example — it lets agents query Prometheus, Loki, Tempo, and dashboards directly.