| name | kanna-telemetry |
| description | Instrument Kanna's code with spans, counters and gauges, and operate the collector that receives them. Read this BEFORE writing any instrumentation — Kanna has one permitted import path for it and a naming convention, so hand-rolled timing code gets rejected by the side-effect lint. Use whenever the task is to measure something: "add a metric", "add a counter", "log performance", "time this", "instrument this", "track how often X happens", "how long does X take", "why is this slow", profiling a turn or a subagent run, memory growth, RSS, heap, an OOM kill, or a pm2 max_memory_restart loop. Use it equally for the collector side: telemetry, tracing, spans, the observability stack, kanna-otel.lowbit.link, kanna-grafana.lowbit.link, Grafana dashboards for Kanna, "why is no telemetry arriving", turning telemetry on or off in settings, or redeploying and debugging the otel-lgtm service on lowbit Dokploy. |
Kanna telemetry — instrument, then observe
Two halves: adding instrumentation to Kanna's code, and operating the collector
that receives it. Start with Instrumenting code if the task is "measure this";
start with Health check if the task is "the telemetry is broken".
Instrumenting code
Domain code imports src/server/observability.ts and nothing else — a pure facade
over @opentelemetry/api exposing withSpan, addCounter, and recordUpDown.
The SDK itself loads only in src/server/otel.adapter.ts, the one file permitted
to import it. That split is what makes instrumentation free to add: with no SDK
registered every facade call is the api package's no-op, so instrumented code needs
no test doubles and costs nothing when telemetry is off. Importing the adapter — or
any @opentelemetry/sdk-* package — from domain code breaks the side-effect seal
and fails lint.
Spans nest through AsyncLocalStorage, so adding depth is a one-line withSpan at
the call site with no handle to thread through the call chain.
Already instrumented, and worth extending rather than duplicating: spans
kanna.turn.start (the spawn pipeline), kanna.subagent.run (a loop's unit of
work), kanna.loop.wake.deliver; counters kanna.subagent.run.finished,
kanna.autocontinue.fired, kanna.queued_message.recovered,
kanna.loop.wake.recovered; and process-memory gauges.
Name new instruments kanna.<area>.<thing> so they group in Grafana alongside
these.
Memory questions specifically
For "RSS is climbing" or an OOM kill, two tools answer different questions and both
are already wired:
KANNA_MEMLOG_MS (default 60000, 0 disables) prints one [kanna/mem] rss=…
line per interval. This is the correlation record — three OOM kills went
undiagnosed for want of exactly this.kill -USR2 <pid> writes a Chrome-DevTools-loadable .heapsnapshot under
<dataDir>/heap-snapshots (KANNA_HEAP_SNAPSHOT=disabled opts out). It is the
only way to answer what holds the bytes on a live process.
The known heavy allocator is transcript parsing: one 96 MB transcript costs ~524 MB
peak RSS to parse (~5.4x its source bytes). Before optimizing, check whether the
path in question loads a whole transcript when a tail read would do.
The collector (lowbit Dokploy)
One collector, many Kanna installs. Each install reports under
service.name = kanna-<machine name> (its computer name), so the Grafana
service dropdown is the inventory of live distributions. Telemetry is a
user-facing Kanna setting (default ON) — "no data" is usually the setting,
not the stack.
Deployed topology (verified 2026-08-14)
| Fact | Value |
|---|
| Dokploy install | lowbit — $LOWBIT_DOKPLOY_BASE_URL, key $LOWBIT_DOKPLOY_API_KEY from ~/.config/dokploy/env (never inline the key) |
| Project | observability (_WxA9mpkAogz9h6hEYaja), env production (-JZ9m8QuozlfkswziqJbr) |
| Compose | otel-lgtm (RjoiChI_VwjX1UBlg5a-j), appName compose-index-auxiliary-circuit-88jw2k, sourceType raw (compose file lives IN Dokploy — edit via compose.update, no git involved) |
| Image | grafana/otel-lgtm:latest, single service otel-lgtm, volumes grafana-data:/data/grafana, prometheus-data:/data/prometheus, loki-data:/data/loki |
| Grafana UI | https://kanna-grafana.lowbit.link (port 3000) — login admin / GF_ADMIN_PASSWORD from the compose env (read it via compose.one, never print it into chat unprompted). The otel-lgtm image ships with ANONYMOUS ADMIN access baked in (GF_AUTH_ANONYMOUS_ENABLED=true — it is a local-dev demo image); the compose overrides it with GF_AUTH_ANONYMOUS_ENABLED: "false" + GF_AUTH_DISABLE_LOGIN_FORM: "false". Keep those overrides in any compose rewrite, or the public URL silently becomes admin-without-login again |
| OTLP ingest | https://kanna-otel.lowbit.link (port 4318, OTLP/HTTP) — POST /v1/traces, POST /v1/metrics |
| Auth on ingest | none (open endpoint) — acceptable for now; keep this in mind before pointing anything sensitive at it |
The general Dokploy API mechanics (env-placeholder rule, deploy verification,
domain creation) live in the sibling dokploy skill — read it for anything
beyond the recipes below.
Kanna side: how an install reports
- Settings group
telemetry: {enabled, endpoint} in ~/.kanna/data/settings.json
(Settings page → "Telemetry Tracing" row). Defaults: enabled: true,
endpoint: https://kanna-otel.lowbit.link. The toggle applies at runtime —
no server restart.
- Service name:
kanna-<sanitized machine display name> (macOS scutil --get ComputerName, else hostname). Raw name rides the host.name resource attribute.
- Env overrides (all optional):
KANNA_OTEL=disabled hard-off,
KANNA_OTEL=enabled force-on, OTEL_EXPORTER_OTLP_ENDPOINT beats the
settings endpoint, KANNA_OTEL_SERVICE_NAME beats the derived name,
KANNA_OTEL_METRIC_INTERVAL_MS (default 15000).
- Precedence logic:
src/server/otel-config.ts (resolveOtelConfig); adapter:
src/server/otel.adapter.ts; ADR adr-20260814-telemetry-settings-gate.
- What arrives: spans
kanna.turn.start, kanna.subagent.run,
kanna.loop.wake.deliver; counters kanna.subagent.run.finished,
kanna.autocontinue.fired, kanna.queued_message.recovered,
kanna.loop.wake.recovered, kanna.turn.tokens, kanna.turn.cost_usd,
kanna.subagent.tokens; histograms kanna.turn.duration_ms,
kanna.subagent.run.duration_ms; gauges kanna.process.rss_bytes,
kanna.process.heap_used_bytes, kanna.process.heap_total_bytes,
kanna.process.external_bytes.
Answering "which install is burning tokens". Turn counts do not answer it —
a 200k-token turn and a 2k-token turn are one turn each. Use the token
counters, whose kind values partition the billed tokens so a bare sum is
the total:
GF_PW=<from compose.one env>
PROM="https://kanna-grafana.lowbit.link/api/datasources/proxy/uid/prometheus/api/v1"
curl -s -u "admin:$GF_PW" --get "$PROM/query" \
--data-urlencode 'query=sort_desc(sum by (job) (increase(kanna_turn_tokens_total[24h])))'
curl -s -u "admin:$GF_PW" --get "$PROM/query" \
--data-urlencode 'query=sum by (job, kind) (increase(kanna_turn_tokens_total[24h]))'
curl -s -u "admin:$GF_PW" --get "$PROM/query" \
--data-urlencode 'query=sum by (job) (increase(kanna_subagent_tokens_total[24h]))'
kanna_turn_cost_usd_total is deliberately SPARSER than the token counters —
PTY-mode turns have no price resolver wired, so a missing cost series means
unknown, never free. Derive spend from tokens × your own rates when it is
absent. Installs older than the release that added these report no token series
at all; that is a version gap, not a quiet install.
Recipes
Always source ~/.config/dokploy/env first. Use curl (python urllib gets
Cloudflare-1010-blocked on lowbit).
Health check (do this first for any "telemetry broken" report):
curl -s -o /dev/null -w '%{http_code}\n' https://kanna-grafana.lowbit.link/api/health
curl -s -o /dev/null -w '%{http_code}\n' -X POST -H 'Content-Type: application/json' \
-d '{}' https://kanna-otel.lowbit.link/v1/traces
curl -s -H "x-api-key: $LOWBIT_DOKPLOY_API_KEY" \
"$LOWBIT_DOKPLOY_BASE_URL/api/docker.getContainersByAppNameMatch?appName=compose-index-auxiliary-circuit-88jw2k"
Which installs are reporting (Grafana datasource proxy → Prometheus):
GF_PW=<from compose.one env>
curl -s -u "admin:$GF_PW" \
"https://kanna-grafana.lowbit.link/api/datasources/proxy/uid/prometheus/api/v1/label/job/values"
Each kanna-<machine> value is one live install. Traces: Grafana → Explore →
Tempo datasource, filter by service name.
Container logs:
curl -s -H "x-api-key: $LOWBIT_DOKPLOY_API_KEY" \
"$LOWBIT_DOKPLOY_BASE_URL/api/compose.readLogs?composeId=RjoiChI_VwjX1UBlg5a-j&containerId=<id from getContainersByAppNameMatch>&tail=200&since=10m"
Redeploy (config/image change): POST /api/compose.deploy with
{"composeId":"RjoiChI_VwjX1UBlg5a-j"}, then VERIFY — fresh row in
deployment.allByCompose AND container uptime reset (Up N seconds).
Dokploy reports success it did not achieve; never trust composeStatus alone.
Edit the compose file (raw source, so the API owns it): read current via
compose.one?composeId=… (composeFile field), modify, POST /api/compose.update
with {"composeId","sourceType":"raw","composeFile":"…"}, then deploy + verify.
Rotate the Grafana admin password: read env via compose.one, replace the
GF_ADMIN_PASSWORD= line, POST /api/compose.saveEnvironment with the WHOLE
env blob (it replaces, not merges), redeploy, verify uptime reset. The
password only reaches the container because the compose file declares
GF_SECURITY_ADMIN_PASSWORD: ${GF_ADMIN_PASSWORD} — keep that placeholder.
Troubleshooting
| Symptom | Likely cause → check |
|---|
| 526 on either host | Missing Traefik router/cert, not a dead panel — first deploy's Let's Encrypt takes ~20 s; check a known-good lowbit host, then domain.byComposeId |
| Ingest 200 but no data in Grafana | The POST {} probe proves routing only. Check the INSTALL: Settings → Telemetry Tracing on? KANNA_OTEL=disabled set? Server log should show [kanna/otel] tracing + metrics enabled with the serviceName |
| One machine missing from service list | That install toggled off, or its machine name changed (new service name). [kanna/mem] lines in its server log confirm the process; the enabled line confirms export |
| Container restart loop | compose.readLogs; volumes persist Grafana/Prometheus/Loki state across restarts, so a wipe means volumes were removed, not restarted |
| All data gone after redeploy | Named volumes survive redeploys. If truly gone, someone removed the volumes — check docker volume ls on the host, and say so rather than guessing |
Safety
- Never echo
$LOWBIT_DOKPLOY_API_KEY or the Grafana password into output.
compose.saveEnvironment replaces the whole blob — read, line-edit, write back.- This stack is shared by every reporting Kanna install; a redeploy drops a few
minutes of incoming telemetry (no queue). Fine to do, but say so.
