investigate-and-fix

name	investigate-and-fix
description	Use when the user reports any error, bug, exception, unexpected behavior, or broken functionality — systematically investigates containers, service logs, database state, code, browser, and traces to find root cause, then fixes it

Investigate and Fix

Systematic investigation and resolution of errors using all available diagnostic tools: container logs, MCP database queries, Playwright browser, code tracing, and the observability stack.

Core principle: Gather evidence from multiple sources IN PARALLEL before forming hypotheses. Never guess — investigate.

REQUIRED BACKGROUND: superpowers:systematic-debugging defines the root-cause methodology. This skill provides the platform-specific investigation toolkit.

Investigation Flow

digraph investigation {
  rankdir=TB;
  "User describes error" [shape=doublecircle];
  "Classify error type" [shape=box];
  "Parallel evidence gathering" [shape=box];
  "Evidence sufficient?" [shape=diamond];
  "Narrow scope, gather more" [shape=box];
  "Form hypothesis" [shape=box];
  "Trace root cause in code" [shape=box];
  "Root cause confirmed?" [shape=diamond];
  "STOP: re-examine assumptions" [shape=octagon, style=filled, fillcolor=red, fontcolor=white];
  "Implement targeted fix" [shape=box];
  "Verify: build + test + reproduce" [shape=doublecircle];

  "User describes error" -> "Classify error type";
  "Classify error type" -> "Parallel evidence gathering";
  "Parallel evidence gathering" -> "Evidence sufficient?";
  "Evidence sufficient?" -> "Narrow scope, gather more" [label="no"];
  "Narrow scope, gather more" -> "Evidence sufficient?";
  "Evidence sufficient?" -> "Form hypothesis" [label="yes"];
  "Form hypothesis" -> "Trace root cause in code";
  "Trace root cause in code" -> "Root cause confirmed?";
  "Root cause confirmed?" -> "Implement targeted fix" [label="yes"];
  "Root cause confirmed?" -> "STOP: re-examine assumptions" [label="no, 2+ failed hypotheses"];
  "STOP: re-examine assumptions" -> "Parallel evidence gathering";
  "Implement targeted fix" -> "Verify: build + test + reproduce";
}

Phase 1: Classify the Error

Signal	Type	Primary investigation
Blank page, wrong UI, JS error	Frontend	Browser snapshot + console + network
4xx/5xx from API	Backend	Service logs + handler code
Data missing or wrong	Data	Database query + handler code
Container won't start / 502	Infrastructure	docker compose ps + container logs
Slow response / timeout	Performance	Traces + DB query plan
Test failure	Test	Test output + handler code
CORS / 301 redirect	Routing	Gateway logs + check trailing slash

Phase 2: Parallel Evidence Gathering

Dispatch multiple investigations simultaneously using subagents or parallel tool calls.

A. Container Health (always check first)

docker compose ps                                    # All containers
docker compose logs --tail=150 <service> 2>&1        # Recent logs
docker compose logs --tail=100 --since=5m <service>  # Last 5 min

B. Service Logs

# Search for errors in a specific service
docker compose logs --tail=500 <service> 2>&1 | grep -iE "error|exception|fail|warn" | tail -50
docker compose logs --tail=200 <service>

C. Database State (via Postgres MCP if available)

Use the mcp__postgres__query (or equivalent) tool against your DB. Schema-qualify tables explicitly if the project uses schema-per-service isolation.

Useful diagnostic queries:

-- Recent records in a table
SELECT * FROM <table> ORDER BY created_at DESC LIMIT 10;

-- If using a transactional outbox (Wolverine, MassTransit, etc.)
SELECT * FROM wolverine_incoming_envelopes ORDER BY id DESC LIMIT 10;

-- Check for orphaned FK references
SELECT * FROM <table> WHERE <fk_column> NOT IN (SELECT id FROM <ref_table>);

D. Browser State (Playwright MCP — for frontend issues)

1. browser_navigate to the affected page
2. browser_snapshot — current DOM tree
3. browser_console_messages — JS errors / warnings
4. browser_network_requests — failed API calls (look for 4xx / 5xx)
5. browser_take_screenshot — visual state

E. Observability — Grafana / Loki / Tempo / Prometheus

If your project has an observability stack via Grafana MCP, use it before falling back to raw docker logs. Investigation order: Logs (Loki) → Traces (Tempo, by TraceId) → Metrics (Prometheus).

Loki — search container logs structurally

# All errors of one service in the last hour
{compose_service="<service-name>", level=~"Error|Fatal"}

# Text match
{compose_service="<service-name>"} |= "BrokenCircuitException"

# Rate of errors over a window (for dashboards)
sum by (compose_service) (count_over_time({level=~"Error|Fatal"}[5m]))

Useful MCP tools (vary by Grafana MCP version):

• list_loki_label_names / list_loki_label_values — discover labels in your environment
• query_loki_logs — run LogQL
• find_error_pattern_logs — auto-detect recurring error patterns

Tempo — TraceQL

# Errored spans of a service
{ resource.service.name="<service>" && status=error }

# Slow server spans (> 1s)
{ resource.service.name=~".*Service" && span.kind=server && duration>1s }

# By HTTP route
{ span.http.route="/api/<entity>/{id}" }

Prometheus — RED + runtime metrics

Adapt to your stack (the metric names below assume OpenTelemetry .NET SDK ≥ 1.10; for other ecosystems substitute equivalents):

# RED — RPS, errors, latency
sum by (service_name) (rate(http_server_request_duration_seconds_count[5m]))
sum by (service_name) (rate(http_server_request_duration_seconds_count{http_response_status_code=~"5.."}[5m]))
histogram_quantile(0.95, sum by (le, service_name) (rate(http_server_request_duration_seconds_bucket[5m])))

# Postgres (postgres-exporter)
sum(pg_stat_database_numbackends)         # active connections
pg_database_size_bytes{datname="<db>"}
sum(increase(pg_stat_database_deadlocks{datname="<db>"}[24h]))

# Redis (redis-exporter)
redis_memory_used_bytes
rate(redis_keyspace_hits_total[5m]) / clamp_min(rate(redis_keyspace_hits_total[5m]) + rate(redis_keyspace_misses_total[5m]), 1)

# RabbitMQ exporter (port 15692)
sum by (queue) (rabbitmq_queue_messages_ready)
rabbitmq_queue_messages{queue="wolverine-dead-letter-queue"}

Use get_dashboard_panel_queries to extract proven PromQL/LogQL out of provisioned dashboards instead of inventing your own.

Why Grafana MCP over docker logs

• Structured LogQL/TraceQL/PromQL vs raw text grep
• Cross-service correlation by TraceId in one place
• Time-range filter, severity filter, JSON parsing built-in
• On prod: docker logs gives the tail of one container without history; Grafana gives retention with search

F. Gateway / Network

docker compose logs --tail=100 nginx 2>&1 | grep -E "502|503|504|405|301"

Trailing slash rule: If your gateway 301s when a request is missing a trailing slash on a collection route, the redirect kills CORS preflight. Always check the exact request URL.

Phase 3: Trace Root Cause in Code

Once you have the error message or stack trace:

1. Grep for the error — find where the message originates
2. Read the handler — understand the full request flow through the vertical slice
3. Check cross-service calls — typed HTTP client in {Service}.Contracts/HttpCommunication/ (or your project's equivalent)
4. Check async messaging — for handler bugs, walk through the routing / outbox / handler triad
5. Check middleware pipeline — auth, error handling, CORS, rate limit

Where to look (adapt to your project layout)

{Service}.Web/                       → Endpoints, middleware, startup, DI
{Service}.Core/                      → Handlers, business logic, domain
{Service}.Contracts/                 → DTOs, events, HTTP client interfaces
{Service}.Infrastructure.<DB>/       → DbContext, migrations, ORM config
Shared/Authentication/               → JWT validation, auth schemes
Shared/Messaging/                    → Integration events, routing

Cross-service auth gotchas

• Token expired? Wrong audience?
• Which auth scheme does the endpoint expect (Cookie vs Bearer)?
• User roles in the auth DB

Phase 4: Fix and Verify

1. Implement targeted fix — single change addressing the root cause, no bundled refactoring
2. Build: project-appropriate command (dotnet build, cargo build, go build, npm run build)
3. Test affected service: project-appropriate command
4. Frontend (if changed): npm run lint && npm run build
5. Reproduce original error — confirm it no longer occurs (browser, API call, or test)

Common Error Patterns

Error	Likely Cause	First Check
401 Unauthorized	Token expired, wrong auth scheme	Auth logs, token claims, cookie vs bearer
403 Forbidden	Missing role	User roles in auth DB
404 Not Found	Wrong route, missing trailing /	Gateway logs, URL format
500 Internal Server Error	Unhandled exception	Service logs (full stack trace)
502 Bad Gateway	Service container down	docker compose ps
CORS error	301 from missing trailing /	Request URL in browser network tab
Connection refused	Container not running or wrong port	Container health, port mapping
Timeout	Slow DB query, deadlock	Traces, EXPLAIN ANALYZE on query
Message-broker error	Queue missing, handler not registered	Broker mgmt UI, framework config
Migration error	Schema drift	Compare snapshot vs DB, re-run migration
OIDC error	Config mismatch	Auth-service OIDC endpoints, frontend env vars

Red Flags — You're Guessing

• Changing code before reading logs
• Assuming the error is in the service the user named (could be upstream)
• Ignoring cross-service communication
• Not checking container health first
• Fixing symptom without tracing to root cause
• Not querying the database when data looks wrong
• Skipping browser console / network when debugging frontend

Parallel Subagent Strategy

For complex issues, dispatch parallel Explore / debugger agents:

Agent	Task
1	Container health + service logs (Bash)
2	Database state for relevant tables (Postgres MCP)
3	Browser state — snapshot, console, network (Playwright MCP)
4	Codebase search for error message / pattern (Grep)

Combine findings into a single evidence picture, THEN trace through code.