archestra-dev-observability

# Archestra Observability Use this skill before changing tracing, metrics, span naming, metric labels, or local observability setup. Run commands from `platform/` unless specifically instructed otherwise. ## Naming new attributes and metrics Before introducing any new span attribute or metric name, look it up — do not coin a name from intuition. - **Span attributes**: search the OTEL semantic-convention registry and use the existing attribute verbatim if one fits. Registry: https://opentelemetry.io/docs/specs/semconv/registry/attributes/gen-ai/ (wider set: https://opentelemetry.io/docs/specs/semconv/registry/attributes/). Example: prompt-cache tokens are `gen_ai.usage.cache_read.input_tokens` and `gen_ai.usage.cache_creation.input_tokens`, not a custom `archestra.usage.*`. Only use an `archestra.*` name when nothing in the registry fits, and say why in a comment. - **"Not yet stable" is not a reason to avoid a standard name.** The whole `gen_ai.*` namespace is Development-stability, including the `gen_ai.usage.*` attributes already emitted here — match that bar, don't custom-namespace to dodge it. - **Metrics**: match the existing `llm_*` / prom-client family and label names in `metrics/`; don't introduce a new metric style. Add a label value to an existing metric only if it won't change what current aggregates mean — otherwise add a dedicated metric (cache tokens use a separate `llm_cache_tokens_total`, not new `type` values on `llm_tokens_total`). ## Local setup ```bash tilt trigger observability docker compose -f dev/docker-compose.observability.yml up -d ``` Both commands are equivalent — the tilt resource wraps the same compose file — and start the full observability stack with pre-configured datasources: Tempo, Loki, OTEL Collector, Prometheus, and Grafana. ## Local URLs - Tempo API: `http://localhost:3200/`. - Grafana: `http://localhost:3002/`. - Prometheus: `http://localhost:9090/`. - Backend metrics: `http://localhost:9050/metrics`. ## Tracing - Follow OTEL GenAI Semantic Conventions (see "Naming new attributes and metrics" — check the registry before adding any attribute): https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-agent-spans/. - LLM spans use `gen_ai.agent.id`, `gen_ai.agent.name`, `gen_ai.provider.name`, `gen_ai.request.model`, `gen_ai.operation.name`, and `archestra.agent.label.<key>` for dynamic agent labels. - MCP spans use `gen_ai.tool.name` and `mcp.server.name`. - Team metadata uses the custom `archestra.<scope>.team.*` namespace (no OTEL registry equivalent), where scope is the principal the teams belong to — `agent` (the executing agent's teams) or `user` (the requesting user's teams). `archestra.<scope>.team.ids` / `.names` are array-valued (a principal can belong to multiple teams), and `archestra.<scope>.team.label.<key>` carries team labels merged per key across the principal's teams. Set via `setTeamAttributes(span, teams, scope)` in `observability/tracing/attributes.ts`; agent teams come from `AgentTeamModel.getTeamLabelInfoForAgent` and user teams from `TeamModel.getTeamLabelInfoForUser`, resolved once per request. - Session tracking uses `gen_ai.conversation.id` from the `X-Archestra-Session-Id` header. - Span names are `chat {model}`, `generate_content {model}`, and `execute_tool {tool_name}`. - Agent label keys are fetched from the database on startup and used as dynamic Prometheus metric label dimensions (see Metrics); the tracing SDK's resource carries only `service.name`/`service.version`. On spans, agent labels are set per-request via `setAgentAttributes`. - Traces are stored in Grafana Tempo. - User identity is tracked with `archestra.user.id`, `archestra.user.email`, and `archestra.user.name` when available. - LLM spans include `archestra.cost` in USD and `gen_ai.usage.total_tokens`. ## Metrics - Prometheus metrics `llm_request_duration_seconds` and `llm_tokens_total` include `provider`, `model`, `agent_id`, `agent_name`, `agent_type`, `source`, and dynamic agent labels as dimensions — deliberately NOT `external_agent_id`, which is client-supplied and unbounded and would explode series cardinality. Do not add it back. - `agent_id` is internal. - `external_agent_id` comes from the client-provided `X-Archestra-Agent-Id` header and is a label only on `agent_runs_total`. - MCP metrics include `agent_id`, `agent_name`, and `agent_type`. - Metrics are reinitialized on startup with current label keys from the database.