opentelemetry

# OpenTelemetry — Core Patterns ## Signal Selection Pick the right signal before writing code: | Signal | Use for | Cost | |---|---|---| | **Traces** | Request lifecycle, latency attribution, distributed causality | High (per-request) | | **Metrics** | Aggregated counts, rates, histograms — dashboards and alerting | Low (pre-aggregated) | | **Logs** | Discrete events with context — errors, audit, debug | Medium | Rule of thumb: metrics answer "how often / how fast", logs answer "what happened", traces answer "why". Don't use traces where metrics suffice. ## Span Design ### Naming convention ``` <verb> <noun> → "fetch user", "send email" <provider>.<operation> → "db.query", "http.get", "cache.set" ``` Never put variable data (IDs, values) in the span name — use attributes. A span name is a cardinality key in your trace index. ### Attributes ```typescript span.setAttributes({ "db.system": "sqlite", "db.operation": "select", "db.sql.table": "users", "user.id": userId, // high-cardinality: OK as attribute, not in name }); ``` Follow [OpenTelemetry Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/) for well-known attribute names (`http.*`, `db.*`, `rpc.*`, etc.) — backends and APMs key off these. ### Events vs child spans - **Event**: instant point-in-time inside the current operation (`span.addEvent("cache_miss")`) - **Child span**: has its own duration, latency matters, useful in trace waterfall ### Status and exceptions ```typescript span.setStatus({ code: SpanStatusCode.ERROR, message: err.message }); span.recordException(err); // adds exception.type / exception.message / exception.stacktrace ``` Call **both** on error. `recordException` alone does not flip status to ERROR — the span appears successful in the UI. ### Guarantee `span.end()` ```typescript // ✓ callback form guarantees end() tracer.startActiveSpan("operation", (span) => { try { return doWork(); } catch (e) { span.recordException(e as Error); span.setStatus({ code: SpanStatusCode.ERROR }); throw e; } finally { span.end(); } }); ``` A span that is never ended leaks in the processor queue and may never export. ## Context Propagation W3C TraceContext (`traceparent` / `tracestate`) is the standard. Inject on outgoing requests, extract on incoming: ```typescript import { propagation, context } from "@opentelemetry/api"; // Outgoing HTTP const carrier: Record<string, string> = {}; propagation.inject(context.active(), carrier); fetch(url, { headers: carrier }); // Incoming (server handler) const ctx = propagation.extract(context.active(), request.headers); tracer.startActiveSpan("handle request", { context: ctx }, (span) => { // span is now a child of the upstream trace }); ``` **Most common bug**: creating a span without extracting the incoming context → the trace waterfall breaks into disconnected root spans. Always extract before starting the root server span. ## Sampling | Strategy | When | |---|---| | `AlwaysOn` | Dev / low-traffic staging | | `TraceIdRatioBased(0.1)` | High-volume production — sample 10% of new traces | | `ParentBased(root: TraceIdRatioBased)` | **Recommended default** — respects upstream decision, samples new roots at ratio | | Tail sampling (OTel Collector) | Need 100% of errors regardless of head-sample decision | `ParentBased` prevents the failure mode where the upstream samples a trace but the downstream drops it (broken waterfall). ## OTLP Exporter Config ```typescript import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http"; import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base"; const exporter = new OTLPTraceExporter({ url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT + "/v1/traces", headers: { Authorization: `Bearer ${process.env.OTEL_API_KEY}` }, }); provider.addSpanProcessor( new BatchSpanProcessor(exporter, { maxQueueSize: 512, scheduledDelayMillis: 2000, exportTimeoutMillis: 10_000, }) ); ``` Use `BatchSpanProcessor` in production — it is async and low-overhead. `SimpleSpanProcessor` blocks the event loop; use only for local debugging. ## Common Pitfalls - **`provider.register()` not called**: SDK is configured but nothing is exported. Call before any instrumentation runs. - **ESM + auto-instrumentation (Node.js)**: `require-in-the-middle` hooks do not fire for ESM static imports. See `otel-node` for the workaround. - **Cloudflare Workers**: no Node.js runtime, fetch-boundary instrumentation needed. See `cloudflare-workers-otel-utels`. - **Span name includes dynamic data**: explodes trace index cardinality. Move to attributes. - **No W3C propagation on outbound calls**: distributed trace breaks — downstream spans appear as orphaned roots.