debug-cloudflare-workers-observability

name

debug-cloudflare-workers-observability

description

Debug deployed Cloudflare Workers using the cf_observability MCP, Wrangler, D1/R2 state, repo evidence, and safe live reproduction. Use for preview/prod runtime incidents involving missing or incomplete logs/traces, webhook delivery, Worker request routing, Workflow runs/events/artifacts, Slack app integrations, Slack API errors such as invalid_arguments, duplicate responses, or cases where the user asks to use Cloudflare Worker Observability or cf_observability to debug c0 behavior.

Debug Cloudflare Workers Observability

Use this as an evidence-first debugging loop for deployed Workers. Start broad enough to prove the request reached the Worker, then narrow through structured logs, durable state, and the owning code boundary.

Core Loop

1. Bound the incident: stage, Worker script name, user-facing URL/path, approximate UTC window, IDs already known, and what safe live action is allowed.
2. Read the repo surface that defines the runtime: Worker name, route, bindings, D1/R2 names, feature flags, and existing observability helpers.
3. Query deployed observability before guessing from code. Use the cf_observability MCP when available; otherwise pivot to Wrangler, D1/R2, app UI, and local reproduction.
4. Cross-check durable state. Logs often prove ingress and routing; D1/R2 usually prove workflow/run/session/artifact state.
5. Reproduce with the smallest safe action only after the path and target environment are clear.
6. Patch the owning boundary. Prefer structured Effect logs/spans/request observers already used by the repo. Do not add noisy catch-all logs or log raw payloads/secrets.
7. Validate locally, deploy only when allowed, then close the loop with a new deployed query and durable-state readback.

cf_observability First Pass

Use tool_search to expose cf_observability if its tools are not already loaded.

Start with observability_keys scoped to the Worker and an absolute ISO timeframe. Avoid relative windows unless you have verified the MCP accepts that exact format.

Common useful fields:

• $metadata.service: Worker script name, such as c0-api-pre.
• $workers.event.request.path: request path for ingress routing.
• $metadata.requestId: Cloudflare request correlation within one invocation.
• $metadata.traceId and annotations.trace.id: useful when populated, but do not assume they are searchable.
• message: structured log text and embedded JSON.
• annotations.*: app-specific metadata from Effect logs/spans.

Load references/cf-observability.md for query JSON, known failure modes, and examples.

Wrangler And State

Before any remote Wrangler work, verify auth without printing secrets:

set -a
source config/.env
set +a
nub exec wrangler whoami

For c0, derive resource names from infra code instead of guessing. In the current pattern, pre resources include c0-db-pre and c0-workflow-artifacts-pre.

Use D1 for durable truth about runs, registrations, sessions, and configuration. Use R2 for workflow artifacts and generated code. Avoid destructive commands unless the user explicitly approved them.

Load references/wrangler-d1-r2.md for command shapes.

Domain Sections

Workflows

Use this section when debugging c0 Workflow triggers, workflow overview rows, run failures, node outputs, generated workflow artifacts, or mismatches between the UI and runtime behavior.

Primary evidence order:

1. Observability ingress/routing logs.
2. workflows, workflow_runs, and workflow_run_events D1 rows.
3. Workflow manifest/code artifacts in R2.
4. Compiled workflow and node implementation code.

Load references/workflows.md for D1 queries, artifact inspection, safe manifest version updates, and how to reason about workflow error logging.

Slack Integration

Use this section when debugging Slack webhook delivery, app mentions, keyword triggers, slash commands, Slack API errors, duplicate Slack replies, or Slack app setup.

Primary evidence order:

1. Query Worker logs for /workflows/slack-apps/ ingress.
2. Confirm channel hydration, registration listing/filtering, matched trigger nodes, and run count.
3. Use D1 run events to decide whether Slack posted once or the workflow/model produced duplicated text.
4. Use browser/Slack only in an explicitly authorized channel, and only after backend targeting is clear.

Load references/slack-integration.md for Slack-specific log names, safe test flow, and invalid_arguments debugging notes.

Reporting

Report the investigation as evidence, not vibes:

• What reached the Worker and how you know.
• Which query or command proved each hop.
• Which query failed or produced incomplete evidence.
• What durable state said.
• The exact owning boundary patched.
• Validation, deploy, and post-deploy readback.