opentelemetry

name	opentelemetry
description	Instrument Flight School with OpenTelemetry traces, metrics, and logs in line with the OTel GenAI semantic conventions and the Aspire dashboard's OTLP receiver.

OpenTelemetry skill

Use this skill whenever you are adding or modifying observability code in Flight School — new spans, new histograms, log enrichment, trace propagation, browser instrumentation, or anything touching the OTLP exporter wiring.

This skill is opinionated: the wider OTel ecosystem offers many ways to do each of these things, and we have picked one. Deviating without a reason creates work for every future contributor.

Provenance

This skill adapts patterns from the following MIT-licensed sources:

• cedricziel/claude-otel-plugin — Next.js + browser OTel patterns.
• oribarilan/97 skills/observability/principles.md — cardinality discipline and golden-signals framing.
• rand/cc-polymath skills/observability/opentelemetry-integration.md — exporter / sampling reasoning.

Authoritative specs cited throughout:

• OTel GenAI semconv: open-telemetry/semantic-conventions-genai — the canonical home as of 2025; the GenAI sections under open-telemetry/semantic-conventions/docs/gen-ai/ are now stubs redirecting here.
• OTel general semconv: open-telemetry/semantic-conventions — resource, error, exception, HTTP.
• @vercel/otel v2.x: vercel/otel — registerOTel options surface.
• Aspire OTel contract: microsoft/aspire src/Aspire.Hosting/OtlpConfigurationExtensions.cs.

No content has been copied verbatim.

How telemetry is wired

Layer	Responsibility	File
Aspire AppHost	Injects OTEL_SERVICE_NAME, OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_RESOURCE_ATTRIBUTES, and short export-interval vars into every child resource on launch.	apphost.ts
Server SDK (web, Next.js)	registerOTel(...) from @vercel/otel installs a tracer provider, a meter provider (via metricReaders), and a log record provider (via logRecordProcessors). The exporters pick up endpoint + protocol from the env vars Aspire injected.	src/instrumentation.ts
Server SDK (worker, standalone Node)	NodeSDK from @opentelemetry/sdk-node started via startWorkerOtel() from inside main() in the worker bootstrap. Must start before any handler import so @opentelemetry/api, undici, and node:http are patched before the route graph loads.	src/worker/lifecycle/otel.ts, started from src/worker/bootstrap.ts
Server semconv	Centralised constants for GenAI attribute keys, metric names, instrumentation-scope names. Do not hard-code these strings anywhere else.	src/lib/observability/semconv.ts
Server tracer/meter API	Wrappers like withSpan, recordAiOperation, recordAiStreamMetrics, recordGitHubOperation, recordJobQueueWait.	src/lib/observability/telemetry.ts
Browser SDK	initBrowserOtel() registers WebTracerProvider + DocumentLoadInstrumentation + FetchInstrumentation; exports JSON over HTTP to a same-origin proxy.	src/lib/observability/browser-otel.ts
Browser → server proxy	Dual same-origin routes that forward OTLP/JSON to the upstream collector: authenticated /api/otel/v1/traces and IP-rate-limited anonymous /api/otel/v1/traces/anonymous. Required because the Aspire OTLP receiver is not CORS-enabled for browsers.	src/app/api/otel/v1/traces/route.ts, src/app/api/otel/v1/traces/anonymous/route.ts, src/app/api/otel/v1/traces/shared.ts
Context propagation	W3C traceparent/tracestate/baggage flow browser → API → worker → Copilot SDK via helpers in src/lib/observability/context-propagation.ts. Always propagate.

Rules

1. Use the right SDK for the runtime

• Web (Next.js): use @vercel/otel's registerOTel in src/instrumentation.ts. Next.js can have edge routes; @opentelemetry/sdk-node is not edge-safe.
• Worker (standalone Node): use NodeSDK from @opentelemetry/sdk-node in src/worker/lifecycle/otel.ts, started from main() in src/worker/bootstrap.ts before the handler graph loads. The worker is not a Next process and has no edge routes; the "no NodeSDK" rule does not apply to it.

2. Server instrumentation is opt-in for metrics and logs

registerOTel({ serviceName }) only installs a tracer provider by default. Metrics and logs must be explicitly wired via the metricReaders and logRecordProcessors options:

registerOTel({
  serviceName: process.env.OTEL_SERVICE_NAME ?? 'flight-school',
  metricReaders: [
    new PeriodicExportingMetricReader({
      exporter: new OTLPMetricExporter(),
    }),
  ],
  logRecordProcessors: [
    new BatchLogRecordProcessor(new OTLPLogExporter()),
  ],
});

If you find yourself creating a meter or a logger and the values never appear in the Aspire dashboard, this is the first thing to check.

3. GenAI semantic conventions are mandatory for AI work

Any code that touches an LLM (Copilot SDK, MCP tool calls, evaluation, hints, authoring) emits spans and metrics using the OpenTelemetry GenAI semconv. The canonical source is open-telemetry/semantic-conventions-genai — the spec was extracted out of the main semconv repo into a dedicated one. Everything in the GenAI namespace is currently Development status: there is no stable surface yet, but the spec is mature enough to ship against and stable enough to commit to in this app.

Concept	Attribute key	Notes
Operation kind	gen_ai.operation.name	chat, generate_content, embeddings, invoke_agent, create_agent, execute_tool, retrieval, plan. Use exactly one of the canonical values.
Requested model	gen_ai.request.model	Exact model name as sent (e.g. gpt-4o).
Response model	gen_ai.response.model	Actual model returned, when available.
Provider	gen_ai.provider.name	e.g. github-copilot, openai, anthropic, aws.bedrock, gcp.vertex_ai. Required on inference spans and metrics.
Token type	gen_ai.token.type	input or output. Required on gen_ai.client.token.usage.
Conversation/session	gen_ai.conversation.id	When available.
Response id	gen_ai.response.id	e.g. chatcmpl-…. Recommended.
Cache tokens (span attrs)	gen_ai.usage.cache_read.input_tokens, gen_ai.usage.cache_creation.input_tokens	Capture on spans alongside the gen_ai.client.token.usage histogram.

Deprecated — do not use:

• gen_ai.system — replaced by gen_ai.provider.name.
• The five legacy log events (gen_ai.system.message, gen_ai.user.message, gen_ai.assistant.message, gen_ai.tool.message, gen_ai.choice) — replaced by the consolidated gen_ai.client.inference.operation.details event. We do not emit prompt/completion bodies today (privacy/cost); if we ever do, use that event with the Opt-In gen_ai.input.messages / gen_ai.output.messages attributes.
• error.message as a metric or span attribute — unbounded cardinality.

Span name: {gen_ai.operation.name} {gen_ai.request.model} (e.g. chat gpt-4o). For agent operations, use the agent name instead of the model: invoke_agent {gen_ai.agent.name}.

Span kind: CLIENT for inference calls (we're calling out to a provider's API). INTERNAL is only correct when the model runs in this process.

Errors: set SpanStatusCode.ERROR and add the stable error.type attribute (the exception's class name, an HTTP status string, or a short bounded code). Do not invent an ai.status attribute. Do not set error.message on metrics — see cardinality rule below.

4. Standard GenAI metrics — names, units, bucket boundaries

All GenAI duration histograms use seconds (s). All token-usage histograms use the synthetic unit {token}.

Metric	Instrument	Unit	When to record
gen_ai.client.operation.duration	Histogram	s	Every AI operation, success or failure. Required.
gen_ai.client.operation.time_to_first_chunk	Histogram	s	First streaming delta arrives. Streaming only; never report on non-streaming calls.
gen_ai.client.operation.time_per_output_chunk	Histogram	s	Each subsequent streaming chunk. Optional but recommended for richer streaming telemetry.
gen_ai.client.token.usage	Histogram	{token}	When the SDK reports usage. Emit once per non-zero token type: input, output, cache_read, cache_write.

Don't confuse time_to_first_chunk with time_to_first_token — the latter is the server-side counterpart and is reserved for LLM serving infrastructure, not clients. Client code always uses time_to_first_chunk.

Recommended histogram bucket boundaries (from the GenAI spec — apply via Views in instrumentation.ts if the SDK defaults don't fit your data):

• Duration / TTFC / time-per-chunk: [0.01, 0.02, 0.04, 0.08, 0.16, 0.32, 0.64, 1.28, 2.56, 5.12, 10.24, 20.48, 40.96, 81.92]
• Token usage: [1, 4, 16, 64, 256, 1024, 4096, 16384, 65536, 262144, 1048576, 4194304, 16777216, 67108864]

Units are seconds, not milliseconds. Do not record *_ms. Convert before recording.

4a. GenAI semconv stability opt-in

The GenAI spec defines OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental as the migration toggle. We are already on the latest conventions (gen_ai.provider.name, consolidated event model), so no opt-in is needed in our code. If a future Copilot SDK update emits the old conventions internally, do not turn this opt-in on globally — wrap or remap at our integration layer instead.

5. Custom app metrics live under flight_school.*

Anything without a standard equivalent uses the flight_school.* prefix: flight_school.jobs.queue_wait, flight_school.ai.stream.delta_count, etc. Document each new metric in this file's "Custom metrics" table when you add one.

6. Cardinality discipline

Attribute keys that go on metrics must have bounded value sets. The following are forbidden as metric labels (they belong on spans/logs):

• user_id, session_id, correlation_id, request_id, trace_id, span_id
• raw URLs containing IDs or query strings — use route templates (e.g. /api/repos/:owner/:repo)
• free-form strings (error messages, user input, model output)
• error.message, exception.message — the spec explicitly deprecates these as metric/span attributes (unbounded)

Spans and log records are sampled and richly-attributed; metrics are aggregated and must stay low-cardinality. The OTel SDK enforces a default cardinality limit of 2000 data points per instrument per collection cycle; once exceeded, additional series are folded into a single otel.metric.overflow=true data point and your dashboards lie. If a metric needs more than ~50 distinct attribute combinations, question the design.

For new public-facing metrics, prefer an explicit allowlist via Views[].attributeKeys in instrumentation.ts to guarantee unknown attributes can't leak in.

7. Instrumentation-scope names

Two scope names exist; both are exported as constants from src/lib/observability/semconv.ts:

• flight-school.observability — server-side tracer and meter.
• flight-school.browser — every browser-side tracer.

Do not pass other strings to trace.getTracer() or metrics.getMeter().

8. Browser instrumentation set is fixed

Use exactly:

@opentelemetry/sdk-trace-web
@opentelemetry/context-zone
@opentelemetry/exporter-trace-otlp-http
@opentelemetry/instrumentation
@opentelemetry/instrumentation-document-load
@opentelemetry/instrumentation-fetch

Do not add @opentelemetry/instrumentation-http, instrumentation-undici, or the auto-instrumentations meta-package on the browser. Do not try to wire @opentelemetry/sdk-metrics on the browser — its browser support is incomplete and we emit no browser-side metrics.

9. Browser exports go to the same-origin proxy

The browser exporter chooses a same-origin proxy route at bootstrap: /api/otel/v1/traces when an Auth.js session cookie exists, otherwise /api/otel/v1/traces/anonymous. The anonymous route is strictly rate-limited per client IP. Never point the browser exporter at the Aspire OTLP endpoint directly — CORS will fail and exposing collector credentials to the browser is unacceptable.

The anonymous route must stay in PUBLIC_PREFIXES in src/proxy.ts. The whole point of the anonymous endpoint is to capture telemetry before a session cookie exists (e.g. on /sign-in); if the auth proxy gates it, unauthenticated page-load traces are silently 401'd and lost — which is the one moment they matter most. The authenticated /api/otel/v1/traces route is intentionally not public and stays gated.

The exporter is configured with fetchOptions: { keepalive: true } so the final batch survives pagehide / beforeunload. Without this the last page.view span and any pending children are silently dropped when the user navigates away or closes the tab.

9a. Browser traces have one root per route: the page.view span

The browser SDK runs a long-lived page.view span that acts as the parent for every fetch and component-mount span emitted while that route is active. Without it, each mount-time useEffect fetch becomes its own root span and the dashboard shows ~10 unrelated traces for a single page load.

How the lifecycle works (see src/lib/observability/route-tracking.ts and src/lib/observability/browser-otel.ts):

1. Module-eval bootstrap. BrowserOtelBootstrap.tsx calls initBrowserOtel() at module-evaluation time (not inside a useEffect). This runs before any component renders, beating React's parent-effects-fire-after-child-effects ordering invariant. A child useEffect fetch would otherwise mis-parent to whatever span happened to be active.
2. History patching. installRouteTracking() monkey-patches history.pushState, history.replaceState, and listens to popstate. App Router's router.push calls pushState, so route transitions are driven synchronously, outside React. Patched functions are tagged with Symbol.for('flight-school.route-tracking.patched') for HMR idempotency.
3. Span lifecycle. A page.view span starts on route enter and ends on the next route change or on pagehide. While it's active, our window.fetch wrapper sets it as the active OTel context before delegating to the original fetch, so FetchInstrumentation reads it as the parent.
4. Visibility vs unload. visibilitychange === 'hidden' calls forceFlush() but does not end the span (the user may return). pagehide ends the span. Mobile browsers don't fire beforeunload reliably — we use pagehide instead.

Don't add a parallel "navigation span" or use useEffect(usePathname) to start a span — both have been tried and both lose the race against child effects.

Don't wrap window.fetch from anywhere except initBrowserOtel. The wrap is idempotent via a Symbol.for marker; a second wrap from another module would either no-op (best case) or stack and double-count (worst case).

Fetch span names use the URL pathname (GET /api/focus not HTTP GET) — set by an applyCustomAttributesOnSpan hook that calls span.updateName(...) after sanitising the URL via extractPathname(). If you add a new browser instrumentation, mirror this pattern.

Live trace-list ergonomics caveat. BatchSpanProcessor only exports ended spans, and page.view deliberately lives for the whole route. Child fetches export first; the parent exports later. The Aspire OTLP collector links them retroactively by traceId, but in the dashboard's live list you'll briefly see children before their parent. This is not a bug — refresh after the route ends and the tree is complete. initBrowserOtel() configures the browser BatchSpanProcessor with maxQueueSize: 100 and scheduledDelayMillis: 5000 (src/lib/observability/browser-otel.ts) to keep browser OTLP export request volume bounded and avoid chatty per-navigation export bursts.

10. Always propagate trace context across boundaries

Every outbound fetch from server code that crosses a service boundary (worker, GitHub MCP, Copilot SDK calls we control) must inject the active traceparent. Use captureTracePropagationHeaders() and mergeTracePropagationHeaders() from context-propagation.ts. Every inbound API route must extract via withExtractedTraceContext().

11. Logs carry trace_id / span_id

Server logs are bridged to the OTel logs API in src/lib/logger.ts via @opentelemetry/api-logs. Because the OTel JS SDK propagates active context through AsyncLocalStorage, trace_id and span_id are attached automatically to every LogRecord emitted from inside an active span — the logger does not (and should not) set them manually on the record.

Don't bypass the logger (no raw console.log in business logic). If you add a new logging sink, route it through the existing Logger class so trace correlation, severity mapping, and OTel bridging are preserved.

The OTel spec is also migrating exception capture from span events into log records (OTEL_SEMCONV_EXCEPTION_SIGNAL_OPT_IN=logs). We do not opt in yet — keep using span.recordException(err) plus error.type. Revisit when the JS SDK marks logs Stable.

12. Aspire env vars are the contract

Aspire's DCP injects these into every child resource automatically when the resource is added via the JavaScript hosting extensions (addNextJsApp, addNodeApp, …). The C# chain WithNodeDefaults → WithOtlpExporter does this transparently — you do not (and cannot) call withOtlpExporter() from the TypeScript AppHost.

Always injected:

• OTEL_SERVICE_NAME — the resource's logical name.
• OTEL_EXPORTER_OTLP_ENDPOINT — gRPC or HTTP, whichever the dashboard exposes.
• OTEL_EXPORTER_OTLP_PROTOCOL — grpc or http/protobuf, matched to the endpoint type. This project uses http/protobuf because aspire.config.json sets ASPIRE_DASHBOARD_OTLP_HTTP_ENDPOINT_URL.
• OTEL_EXPORTER_OTLP_HEADERS — includes x-otlp-api-key=… when the dashboard runs in API-key auth mode (default).
• OTEL_RESOURCE_ATTRIBUTES — at minimum service.instance.id={uuid}.

Injected only in IsDevelopment() mode:

• OTEL_BSP_SCHEDULE_DELAY=1000 — span batch flush every 1s instead of 5s.
• OTEL_BLRP_SCHEDULE_DELAY=1000 — log batch flush every 1s.
• OTEL_METRIC_EXPORT_INTERVAL=1000 — metric export every 1s instead of 60s.
• OTEL_TRACES_SAMPLER=always_on — sample everything in dev.
• OTEL_METRICS_EXEMPLAR_FILTER=trace_based — emit metric exemplars linking to active spans (the dashboard renders these as clickable dots on metric charts).

Read only the variables in the "always injected" list from app code. Do not invent app-specific env vars to route telemetry. If Aspire's contract isn't enough, fix it in apphost.ts via withEnvironment(...).

13. Known Next.js + OTel gotchas

• The experimental.instrumentationHook flag is gone in Next.js ≥ 15 — do not set it.
• Duplicate fetch spans: @vercel/otel v2's default instrumentations: ["auto"] installs FetchInstrumentation, and Next.js's built-in tracer also emits AppRender.fetch spans for every server-side fetch(). Set NEXT_OTEL_FETCH_DISABLED=1 on the Next.js web resource in apphost.ts (via withEnvironment) to suppress Next.js's built-in span and keep only the richer FetchInstrumentation span. Without this every server fetch is double-counted. The worker is no longer a Next.js app, so this var is intentionally absent from the worker resource.
• App Router route handlers (app/api/.../route.ts) automatically set http.route. Pages Router and middleware do not — set it explicitly if you ever add one of those.
• Resource attribute merge order: OTEL_RESOURCE_ATTRIBUTES (from Aspire, picked up by the env detector) wins over the Vercel defaults @vercel/otel adds, which win over the attributes field you pass to registerOTel. If a vercel.* attribute needs to disappear, drop it explicitly.

14. Resource attributes we set

The Aspire env detector populates service.name and service.instance.id from OTEL_SERVICE_NAME / OTEL_RESOURCE_ATTRIBUTES. On top of that, ensure these on the server resource:

• service.version — read from INSTRUMENTATION_SCOPE_VERSION (which tracks the app version).
• deployment.environment.name — note the .name suffix (the old deployment.environment is deprecated). Read from NODE_ENV or set explicitly in apphost.ts per environment.

The browser resource sets service.name=flight-school-browser and service.version so the dashboard groups browser telemetry as its own resource alongside the server.

14a. Service-tier naming — how to tell who emitted a span

A trace can originate in three logical tiers, but only two of those tiers map cleanly to a process. The Next.js process serves both page renders and API route handlers — same Node runtime, same OTel SDK, same service.name. That's a fact of Next.js, not a deficiency.

Logical tier	What it does	How to identify it in a trace
Browser	All client JS — fetches, navigations, document load	service.name=flight-school-browser (resource attribute)
API layer	App Router route handlers under app/api/**/route.ts	service.name=flight-school-web AND http.route starts with /api/
Page render	App Router pages, RSC, server components	service.name=flight-school-web AND http.route does not start with /api/
Worker	Background queue consumer	service.name=flight-school-worker

Recommended dashboard filters:

• "All browser activity" → filter service.name = flight-school-browser.
• "All API calls" → filter service.name = flight-school-web and http.route =~ ^/api/.
• "All page renders" → filter service.name = flight-school-web and http.route !~ ^/api/.
• "All worker activity" → filter service.name = flight-school-worker.

Why we don't split flight-school-web into two services. It's genuinely one Node process. The OTel HTTP semconv puts http.route on every server span specifically so dashboards can slice the same service along route patterns without inventing fake services. Splitting would require either a second Next.js process (doubles cold-start cost) or lying about reality (one process, two service.names — confuses exemplar linking and resource queries). http.route is the paved path.

Why we don't (currently) add a flight_school.tier span attribute. Tempting, but it would duplicate http.route — the dashboard can already filter on the prefix. The day we want tier in metric labels (low-cardinality counters of API vs page work), that's the point to add a Views mapping that derives flight_school.tier ∈ {api, page} from http.route and drops the raw route, since the raw route is unbounded cardinality for metrics. Don't add it speculatively.

Naming nit. flight-school-web is slightly ambiguous — it's the Next.js process, not "the website". If a future rename makes things clearer, flight-school-nextjs is the better service name. Until then, remember: web = "the Next.js process that does both pages and API", filtered apart by http.route.

15. Browser telemetry: traces only, by design

The same-origin proxy routes (/api/otel/v1/traces and /api/otel/v1/traces/anonymous) forward traces only. Browser-side metrics and logs are intentionally not collected:

• The OTel JS SDK metrics surface is incomplete on the browser.
• Most browser-side counters we'd want (page views, errors, user interactions) are better captured as span events on document-load and navigation spans, which we already emit.
• Forwarding browser logs would mean either trusting client-supplied trace context or breaking trace correlation — neither is acceptable.

If you need a new browser metric, prefer adding a span event or attribute to an existing browser span. Do not wire sdk-metrics or sdk-logs on the browser without a written design.

16. Suppress the OTel-proxy self-tracing feedback loop

The browser BSP flushes pending spans on document.visibilitychange, so every tab switch POSTs to /api/otel/v1/traces. Without intervention, @vercel/otel's server-side auto-instrumentation traces that POST, producing four spans per export (POST, POST /api/otel/v1/traces, resolve page components, executing api route (app) /api/otel/v1/traces). On a quiet app these self-spans drown out real telemetry on the dashboard.

The fix is a traceSampler that drops spans whose name or attributes identify the proxy route. The unsampled root propagates NOT_RECORD to its children via the standard parent-based chain, so all four spans disappear together:

// src/instrumentation.ts
import { createTelemetryHygieneSampler } from '@/lib/observability/proxy-sampler';

registerOTel({
  // ...
  traceSampler: createTelemetryHygieneSampler(),
});

The sampler matches the proxy-route prefix (/api/otel/v1/traces) across spanName, http.target, url.path, and http.route because those attributes are populated at different stages of the request lifecycle (HTTP instrumentation sets http.target / url.path at span creation; Next.js sets http.route later, once the route has matched).

Do not "fix" this by disabling the browser BSP's visibilitychange flush — losing spans when a tab is backgrounded is worse than a few self-spans, and the sampler closes the loop completely without that trade-off.

Red flags to call out in review

• A new tracer.startSpan without try/finally/span.end().
• A new metric name not under gen_ai.*, flight_school.*, or the OTel HTTP/RPC/DB semconv.
• Any setAttribute('ai.*', …) — should be gen_ai.*.
• Any gen_ai.system attribute — deprecated; use gen_ai.provider.name.
• Any of the deprecated GenAI log events (gen_ai.system.message, gen_ai.user.message, etc.) — use gen_ai.client.inference.operation.details if we ever capture bodies.
• A duration metric or attribute with _ms suffix — units must be seconds.
• A metric label that could equal userId, correlationId, traceId, a raw URL, error.message, or any unbounded string.
• A new trace.getTracer('something-custom') — use the scope constants from semconv.ts.
• An outbound fetch (server-side) that doesn't merge the propagation headers.
• A new logger call via console.* directly.
• Anything that reads process.env.OTEL_* outside instrumentation.ts or the OTLP proxy route — the SDK consumes those; app code shouldn't.

Custom metrics registry

When you add a new flight_school.* metric, append a row here:

Metric	Instrument	Unit	Attributes (and expected cardinality)	Defined in
flight_school.jobs.queue_wait	Histogram	s	job.type (bounded)	telemetry.ts
flight_school.ai.stream.delta_count	Histogram	{delta}	gen_ai.request.model, ai.mcp_enabled, ai.pool_hit, ai.stream.terminal_state	telemetry.ts
flight_school.ai.stream.delta_bytes	Histogram	By	same as above	telemetry.ts
flight_school.ai.stream.tool_calls	Counter	{call}	same as above	telemetry.ts

Smoke checklist after non-trivial OTel changes

1. aspire run and open the dashboard.
2. Metrics view — every metric you touched is present with the expected name and unit.
3. Traces view — exercise the changed code path; spans carry the GenAI attributes; no duplicate fetch spans.
4. Structured Logs view — recent logs carry a trace_id/span_id that matches a trace.
5. npm test passes — telemetry unit tests assert names and attribute keys.