archestra-dev-observability

name	archestra-dev-observability
description	Use when changing Archestra tracing, metrics, OpenTelemetry, Tempo, Grafana, Prometheus, LLM/MCP spans, observability labels, or local observability setup.

Archestra Observability

Use this skill before changing tracing, metrics, span naming, metric labels, or local observability setup.

Run commands from platform/ unless specifically instructed otherwise.

Naming new attributes and metrics

Before introducing any new span attribute or metric name, look it up — do not coin a name from intuition.

• Span attributes: search the OTEL semantic-convention registry and use the existing attribute verbatim if one fits. Registry: https://opentelemetry.io/docs/specs/semconv/registry/attributes/gen-ai/ (wider set: https://opentelemetry.io/docs/specs/semconv/registry/attributes/). Example: prompt-cache tokens are gen_ai.usage.cache_read.input_tokens and gen_ai.usage.cache_creation.input_tokens, not a custom archestra.usage.*. Only use an archestra.* name when nothing in the registry fits, and say why in a comment.
• "Not yet stable" is not a reason to avoid a standard name. The whole gen_ai.* namespace is Development-stability, including the gen_ai.usage.* attributes already emitted here — match that bar, don't custom-namespace to dodge it.
• Metrics: match the existing llm_* / prom-client family and label names in metrics/; don't introduce a new metric style. Add a label value to an existing metric only if it won't change what current aggregates mean — otherwise add a dedicated metric (cache tokens use a separate llm_cache_tokens_total, not new type values on llm_tokens_total).

Local setup

tilt trigger observability
docker compose -f dev/docker-compose.observability.yml up -d

Both commands are equivalent — the tilt resource wraps the same compose file — and start the full observability stack with pre-configured datasources: Tempo, Loki, OTEL Collector, Prometheus, and Grafana.

Local URLs

• Tempo API: http://localhost:3200/.
• Grafana: http://localhost:3002/.
• Prometheus: http://localhost:9090/.
• Backend metrics: http://localhost:9050/metrics.

Tracing

• Follow OTEL GenAI Semantic Conventions (see "Naming new attributes and metrics" — check the registry before adding any attribute): https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-agent-spans/.
• LLM spans use gen_ai.agent.id, gen_ai.agent.name, gen_ai.provider.name, gen_ai.request.model, gen_ai.operation.name, and archestra.agent.label.<key> for dynamic agent labels.
• MCP spans use gen_ai.tool.name and mcp.server.name.
• Team metadata uses the custom archestra.<scope>.team.* namespace (no OTEL registry equivalent), where scope is the principal the teams belong to — agent (the executing agent's teams) or user (the requesting user's teams). archestra.<scope>.team.ids / .names are array-valued (a principal can belong to multiple teams), and archestra.<scope>.team.label.<key> carries team labels merged per key across the principal's teams. Set via setTeamAttributes(span, teams, scope) in observability/tracing/attributes.ts; agent teams come from AgentTeamModel.getTeamLabelInfoForAgent and user teams from TeamModel.getTeamLabelInfoForUser, resolved once per request.
• Session tracking uses gen_ai.conversation.id from the X-Archestra-Session-Id header.
• Span names are chat {model}, generate_content {model}, and execute_tool {tool_name}.
• Agent label keys are fetched from the database on startup and used as dynamic Prometheus metric label dimensions (see Metrics); the tracing SDK's resource carries only service.name/service.version. On spans, agent labels are set per-request via setAgentAttributes.
• Traces are stored in Grafana Tempo.
• User identity is tracked with archestra.user.id, archestra.user.email, and archestra.user.name when available.
• LLM spans include archestra.cost in USD and gen_ai.usage.total_tokens.

Metrics

• Prometheus metrics llm_request_duration_seconds and llm_tokens_total include provider, model, agent_id, agent_name, agent_type, source, and dynamic agent labels as dimensions — deliberately NOT external_agent_id, which is client-supplied and unbounded and would explode series cardinality. Do not add it back.
• agent_id is internal.
• external_agent_id comes from the client-provided X-Archestra-Agent-Id header and is a label only on agent_executions_total.
• MCP metrics include agent_id, agent_name, and agent_type.
• Metrics are reinitialized on startup with current label keys from the database.