live-debug

# Live debugging `just run_local` and `just stack up` start the LGTM collector by default and the agent browser with `--with-chrome` (both global: one per machine, shared across instances, left running): | Endpoint | What | | --- | --- | | http://localhost:3001 | Grafana (anonymous admin) — traces + logs UI | | http://localhost:3200 | Tempo API — traces | | http://localhost:3100 | Loki API — service logs | | http://localhost:9090 | Prometheus API — metrics | | localhost:4317 / 4318 | OTLP intake (gRPC / HTTP), alias `otel-collector` | | http://localhost:9222 | Headless Chrome (CDP) | `--traces off` disables the collector; `--traces jaeger|datadog` swaps it. All collectors bind 4317/4318 — exactly one runs at a time. Every Rust service exports spans AND its `tracing` events (as correlated log records) over OTLP. The frontend exports browser spans through the proxy and propagates `traceparent`, so one trace covers browser → proxy → services. Wiring happens at stack start: if you start a collector by hand, restart the stack to pick it up. ## Query telemetry (prefer APIs over the Grafana UI) The `grafana` MCP server (`.mcp.json` / `opencode.json` / `.cursor/mcp.json`, Docker `mcp/grafana` on the host network) is pointed at this Grafana. Prefer its tools: - Logs: `query_loki_logs`, `list_loki_label_values`, `find_error_pattern_logs` - Traces: `tempo_traceql-search`, `tempo_get-trace`, `tempo_get-attribute-values`, `tempo_docs-traceql` (proxied from Tempo's own MCP server; every `tempo_*` call needs `datasourceUid: "tempo"`) - Metrics: `query_prometheus`; plus `search_dashboards`, `generate_deeplink` Datasource UIDs are stable: `loki`, `prometheus`, `tempo`, `pyroscope`. Typical calls — service names are the binary names (`email_service`, `document-storage-service`); match on route, duration, or any span attribute: - `tempo_traceql-search` `{datasourceUid: "tempo", query: '{resource.service.name="email_service" && status=error}'}` (also `'{span.http.route="/documents" && duration>500ms}'`), then `tempo_get-trace` with the returned trace ID. Searches default to the past hour; widen with RFC3339 `start`/`end`. - `query_loki_logs` `{datasourceUid: "loki", logql: '{service_name="email_service"} |= "error"'}` — log lines carry `trace_id`/`span_id` for correlation. Discover services with `list_loki_label_values` on `service_name`. Timing quirks: Tempo's search index flushes every ~30s — a trace you just produced is fetchable by ID immediately but may not show in search yet. Everything above is also plain HTTP: TraceQL search at `http://localhost:3200/api/search?q=<traceql>`, trace fetch at `/api/traces/<id>`, LogQL at `http://localhost:3100/loki/api/v1/query_range?query=<logql>` (`start`/`end` are unix epoch **nanoseconds**). Use the HTTP form when there is no MCP (pi), and for bulk retrieval you want to reduce before reading — a full trace can be 50+ spans, so `curl /api/traces/<id> | jq` (filter to slow spans, compute offsets) beats dumping `tempo_get-trace` output into context. `docker compose -p macro logs -f <service>` still works for raw stdout, but Loki is queryable and survives restarts. Verbosity knobs (set in the shell before `just run_local`, or per service in Doppler): `RUST_LOG` filters console + Loki output; `OTEL_TRACE_FILTER` independently filters exported spans (default `info`). Lowering `RUST_LOG` never silences traces. ## Drive the shared headless Chrome `--with-chrome` runs a headless Chrome in Docker with CDP on 9222 (if 9222 doesn't answer, start it: the compose command is in the `headless-chrome` comment in `docker/docker-compose.yml`). The `chrome-devtools` MCP server (`.mcp.json` / `opencode.json` / `.cursor/mcp.json`) is already pointed at it — prefer its tools (navigate, snapshot, click, evaluate, console, network) for browser work. State (cookies, login) persists across agent sessions until the container restarts. - The container uses host networking, so plain localhost URLs work: the app is **http://localhost:3000/app**, the proxy `https://localhost:8090` (named instances remap these ports; read the stack summary). - A human can watch the browser live at **http://localhost:6080/vnc.html** (noVNC over the Xvfb display) — work in the visible window, not isolated contexts, when someone may be watching. - Login is passwordless: any email works, and the login API returns the code in its response (also visible in Mailpit at http://localhost:8025). - From Playwright instead: `chromium.connectOverCDP('http://localhost:9222')`. For token-injection and route-interception recipes see `apps/web/docs/playwright-debugging.md`. Correlate a browser repro with backend traces: note the time, then search Tempo for that window — the browser's `traceparent` means the frontend action and the Rust handler share one trace ID. ### chrome-devtools technique - Snapshot-first: `take_snapshot` after every navigation or pane change; act only on uids from the latest snapshot (uid prefixes bump on re-render). Macro snapshots are huge — split panes duplicate the doc text — so save big ones to a file (`filePath` param) and grep them. - `navigate_page` can time out while the SPA actually loaded (cold Vite compile); follow with `wait_for` on expected text instead of re-navigating. - `wait_for` matches any text presence, including placeholders. For "AI finished"-style conditions, poll in `evaluate_script` for the Stop button's absence — the only reliable completion signal. - `fill` works on plain inputs but NOT contenteditable: click to focus, then `type_text` (Enter splits paragraphs/sends). Combobox token fields need type → wait for the "N options available" live region → Enter to tokenize. - If a radio/tab control won't click ("did not become interactive"), click its adjacent label text node instead. - On any error dialog or blank state: `list_console_messages` + `list_network_requests` (filter xhr/fetch), then `get_network_request` for the failing request's body — pairing console error with failing request localizes the fault in one step. - Verify editor state with `evaluate_script` (e.g. query `[contenteditable] strong` to confirm an AI edit) — cheaper and more precise than screenshots. ### Driving the Macro app The condensed version is below; the full field-tested guide (routes, every surface, keyboard model, crash recovery, trace correlation from a network request's `traceparent`) is `docs/AGENT_GUIDE/`. - Unauthenticated users land on `/app/welcome`: "Continue with email" → fill the email input → "Continue". Locally this may log in with no code prompt; otherwise the code is in Mailpit. First login auto-creates the user. - Documents live at `/app/md/<uuid>`; a doc-scoped AI chat at `/app/md/<uuid>/chat/<chatId>`; split panes give the right pane its own URL segment (`/app/md/<uuid>/channel/<channelId>`). - Everything is created via the top-left "Create" button (Document D, Channel G, Message M, Task T, …). Sidebar buttons are named "Go to X" in the a11y tree. Search is the "Search" button — results appear as you type, no Enter. - Editor: title field is focused on creation; type the title, Enter moves into the body. The contenteditable's a11y `value` exposes the full body text, so snapshots double as content verification. - AI edit: "Edit with AI" button under the editor → type the instruction → Enter. Edits apply in place; done when the "Stop" button disappears. AI chat: "Ask Macro" in the doc's Actions panel (doc pre-attached), Enter sends, "Stop generating" disappears when the response is complete. - Channels: "Create" → Channel → name it, tokenize invitees in the "To:" combobox, "Create Channel". Verify membership in the Participants tab. Composer: Enter sends; the "Task" switch turns a message into a task. ## Write tracing code that is debuggable Follow CLAUDE.md's tracing rules (`err` on `Result`-returning `#[instrument]`, never `level = "info"`, `tracing::error!(error=?e, "msg")`, prefer `.inspect_err`). Beyond those: - Instrument boundaries, not plumbing: HTTP handlers get spans from `macro_tower_layers`; add `#[tracing::instrument]` to queue consumers, cross-service client calls, and multi-step business operations — the places a trace would otherwise go dark. - Skip bulky args (`#[instrument(skip(payload), fields(document_id = %id))]`) and record the IDs you will actually search by: entity IDs, user IDs, counts. A span you can't find by ID is a span you can't use. - Record late-known values with `tracing::Span::current().record(...)` rather than emitting a second event. - Events inside a span inherit its trace: one `tracing::warn!` with fields beats three unstructured `debug!`s. Fields, not format strings — `warn!(attempts, "retrying")`, not `warn!("retrying attempt {attempts}")`. - Verify your instrumentation live: run the code path, then confirm the span shows up in Tempo with the fields you expect. Unverified instrumentation is the usual reason "the trace was empty" during a real incident.