Implementing distributed tracing with W3C Trace Context — the byte-precise `traceparent` and `tracestate` header formats, OpenTelemetry's W3CTraceContextPropagator as the modern default (replacing X-B3-* and X-Datadog-*), head vs tail sampling, and HTTP→SQL trace propagation via sqlcommenter. Grounded in the W3C REC and OpenTelemetry specs. NOT for OTel metrics/logs pipelines, vendor backend setup (Datadog/Honeycomb/Jaeger/Tempo), or browser RUM tracing.
Implementing distributed tracing with W3C Trace Context — the byte-precise `traceparent` and `tracestate` header formats, OpenTelemetry's W3CTraceContextPropagator as the modern default (replacing X-B3-* and X-Datadog-*), head vs tail sampling, and HTTP→SQL trace propagation via sqlcommenter. Grounded in the W3C REC and OpenTelemetry specs. NOT for OTel metrics/logs pipelines, vendor backend setup (Datadog/Honeycomb/Jaeger/Tempo), or browser RUM tracing.
{"category":"Observability","tags":["observability","tracing","opentelemetry","w3c-trace-context","traceparent","sampling","sqlcommenter"],"pairs-with":[{"skill":"logging-observability","reason":"Trace IDs propagated here become the correlation keys stamped into structured logs so a slow span links to its log lines."},{"skill":"observability-apm-expert","reason":"Owns the APM backend and dashboards that consume the sampled traces this skill's propagation and sampling design produces."},{"skill":"monitoring-stack-deployer","reason":"Deploys the OTel Collector where tail-sampling and probabilistic-sampler processors from this skill's recipes actually run."}],"provenance":{"kind":"first-party","owners":["port-daddy"]},"io-contract":{"kind":"deliverable","consumes":["[Truncated]","[Truncated]"],"produces":["[Truncated]","[Truncated]"]}}
Distributed Tracing with W3C Trace Context
TL;DR: traceparent is a 55-byte header with a fixed format (version-trace_id-parent_id-flags, lowercase hex). It's the modern default for cross-service trace propagation, replacing per-vendor X-B3-*, X-Datadog-*, etc. OpenTelemetry's W3CTraceContextPropagator is the canonical implementation. Decide head vs tail sampling early — they're not interchangeable. For HTTP→SQL correlation, append the trace context as a SQL comment via sqlcommenter format.
Jump to your fire
Symptom
Section
"Traces break across our Node→Go service boundary"
flowchart TD
A[Setting up distributed tracing] --> B{Existing tracing infra?}
B -->|None| C[Default: OpenTelemetry SDK<br/>+ W3CTraceContextPropagator]
B -->|Has B3/Jaeger/X-Ray headers| D[Configure composite propagator<br/>W3C + legacy for transition]
C --> E[Decide sampling strategy]
D --> E
E --> F{Need to filter traces by trace-wide attribute<br/>e.g. always sample errors?}
F -->|No, simple % sampling| G[Head sampling<br/>at SDK ParentBased+TraceIDRatio]
F -->|Yes| H[Tail sampling<br/>at Collector<br/>tailsamplingprocessor]
G --> I{Need to correlate to DB queries?}
H --> I
I -->|Yes| J[Enable sqlcommenter<br/>OTel SQL instrumentation MAY-flag]
I -->|No| K[Done]
"Vendors MUST expect the header name in any case (upper, lower, mixed), and SHOULD send the header name in lowercase." (§3.2.1)
"If the trace-id value is invalid (for example if it contains non-allowed characters or all zeros), vendors MUST ignore the traceparent." (§3.2.2.3)
"Vendors MUST ignore the traceparent when the parent-id is invalid (for example, if it contains non-lowercase hex characters)." All-zeros parent-id is also invalid. (§3.2.2.4)
"A vendor receiving a traceparent request header MUST send it to outgoing requests." "The parent-id field MUST be set to a new value with the sampled flag update." (§3.4)
"Vendors MUST NOT parse or assume anything about unknown fields for this version." (§3.2.4)
Sampled flag (§3.2.2.5.1)
The current spec (version 00) defines one flag bit:
01 = sampled (record + export); 00 = not sampled. The flag may flip when parent-id is updated but should reflect the upstream sampling decision unless the local sampler overrides explicitly.
If pre-configured, Propagators SHOULD default to a composite Propagator containing the W3C Trace Context Propagator and the Baggage Propagator specified in the Baggage API.
OpenTelemetry deprecated Jaeger and OT Trace propagators in favor of W3C ("use the W3C TraceContext instead"). B3 is still maintained as a backward-compat option.
Inject sends both header sets; extract reads whichever arrives. Run the composite propagator until the last legacy upstream is migrated, then drop the B3 entry.
Sampled: A trace or span is processed and exported. … Not sampled: A trace or span is not processed or exported.
Head sampling
Head sampling is a sampling technique used to make a sampling decision as early as possible. A decision to sample or drop a span or trace is not made by inspecting the trace as a whole.
The most common form of head sampling is Consistent Probability Sampling. This is also referred to as Deterministic Sampling. In this case, a sampling decision is made based on the trace ID and the desired percentage of traces to sample.
Pros: cheap, deterministic across services (same trace-id everywhere → same decision), runs in the SDK.
Con (verbatim): "It is not possible to make a sampling decision based on data in the entire trace. For example, you cannot ensure that all traces with an error within them are sampled with head sampling alone."
Tail sampling
Tail sampling is where the decision to sample a trace takes place by considering all or most of the spans within the trace.
Use cases: always-sample-on-error, latency-based sampling, attribute-based sampling, differential rates per service.
Cons: stateful (the collector must hold all spans for a trace until it decides), vendor-specific tooling, expensive.
The pragmatic recipe
Need
Strategy
Bulk volume reduction
Head sampling (e.g., ParentBased(TraceIDRatio(0.1))) — 10% baseline
Always sample errors
Tail sampling at the OTel Collector with tailsamplingprocessor
Always sample slow requests (p99+)
Tail sampling on latency policy
Always sample for a specific tenant
Both: head sampling 100% via ParentBased, ratio low elsewhere
The sampled flag is the wire-level signal: when a head sampler in service A decides "sample this trace," service B inherits the bit via traceparent flags and respects it (per W3C §3.4). This is what makes head sampling consistent across services.
4. HTTP→SQL trace propagation via sqlcommenter
A common debugging gap: the trace shows a slow service span, but you can't tell which SQL query was slow because the DB log doesn't know about traces. Sqlcommenter solves this by appending the trace context as a SQL comment.
Instrumentations MAY propagate context using SQL commenter by injecting comments into SQL queries before execution. SQL commenter-based context propagation SHOULD NOT be enabled by default, but instrumentation MAY allow users to opt into it.
The instrumentation implementation SHOULDappend the comment to the end of the query.
The tracestate field value is a list of list-members separated by commas (,). A list-member is a key/value pair separated by an equals sign (=).
There can be a maximum of 32 list-members in a list.
Identifiers MUST begin with a lowercase letter or a digit, and can only contain lowercase letters (a-z), digits (0-9), underscores (_), dashes (-), asterisks (*), and forward slashes (/).
The leftmost entry is the most-recent vendor. When mutating, prepend your entry; if you'd exceed the limit, drop the rightmost (oldest) first.
If the vendor failed to parse traceparent, it MUST NOT attempt to parse tracestate. Note that the opposite is not true: failure to parse tracestateMUST NOT affect the parsing of traceparent.
Anti-patterns
Anti-pattern
Why it bites
Fix
Generating uppercase hex in trace-id
Receiving vendors MUST ignore the header → trace breaks at boundary
Always lowercase; use SDK helpers, never string-format manually
Reusing parent's parent-id instead of allocating a new one per service
Trace tree collapses; can't tell which span is which
Always generate a new parent-id (= span_id) per local span
Start at 1-10%; increase only for specific routes/tenants
Mixing W3C and B3 propagators without a composite
Some hops drop the trace
Composite propagator until migration complete
Using vendor SDK instead of OTel
Lock-in; can't switch backends without re-instrumenting
OTel SDK + vendor exporter
Putting PII in tracestate
Headers logged at every hop; cardinality explosion
tracestate is for trace-routing data only; user attributes go in span attributes (filtered)
Leaving SQL commenter on by default
Plan-cache impact on MySQL/Oracle/SQL Server
Opt-in per the OTel SHOULD-NOT-default rule
Novice / Expert / Timeline
Novice
Expert
Adding tracing
Vendor agent (Datadog, NewRelic)
OTel SDK + vendor exporter; portable
Cross-service propagation
Hopes vendor SDK handles it
Verifies traceparent in HTTP captures; tests at boundaries
Sampling
100% in dev, panic in prod
Head sampling baseline + tail sampling for errors/slow
DB correlation
Reads slow query log + guesses
sqlcommenter on; click slow query → trace
Multi-vendor migration
Picks one, all-or-nothing
Composite propagator; gradual rollout
Timeline test: a request that errors in service C — can you find the trace that includes spans from A → B → C and the SQL queries that ran in B? Expert answer: yes, in seconds, via tail-sampled error trace + sqlcommenter linkage. Novice answer: not really; you correlate timestamps by hand.
Quality gates
A tracing change ships when:
Test: A request crosses every service boundary preserving traceparent — end-to-end test asserts the same trace_id appears in spans from each service.
Test:traceparent headers in test fixtures are exactly 55 chars, lowercase hex, with the correct field separators.
Test: Sampling decision is deterministic across services for the same trace_id (verified by replaying the same request and checking sampled-or-not is consistent).
Test (tail sampling, if used): Errors are sampled at 100% — synthetic error injection produces a trace in the backend.
Test (sqlcommenter, if enabled): A slow query log line includes the traceparent comment, and the trace_id matches the HTTP span that triggered it.
Test: Composite propagator (during migration) correctly extracts from both W3C and legacy headers — fixture for each shape.
Manual: Trace storage cost projection sized for current sampling rate × request volume.
Deterministic Audit
Before wiring (or reviewing) a propagation + sampling design, write it as a
JSON plan matching schemas/trace-propagation-plan.schema.json and run the
auditor:
auditTracePropagation(plan) (in scripts/trace_propagation_audit.mjs) turns
this skill's Quality Gates and Anti-patterns into machine-checkable rules over
structured fields — no keyword matching: a reused parent-id across services
(critical — collapses the trace tree), unvalidated traceparent format (uppercase
hex → MUST-ignore), legacy upstreams without a composite propagator, a
vendor-proprietary propagator, head sampling when error traces must always be
kept, tail sampling with no head-sampling safety net, 100% production sampling,
default-on sqlcommenter and its MySQL/Oracle/SQL Server plan-cache impact, and
PII in tracestate (critical). It returns
{ pass, score, findings, recommendations }. examples/sample-input.json is a
mid-migration composite-propagator plan with head+tail sampling (pass: true,
zero findings).
NOT for this skill
Metrics or logs in OpenTelemetry (use opentelemetry-metrics-design, opentelemetry-logs-design)
Specific backend setup (Datadog, Honeycomb, Jaeger, Tempo) (use vendor-specific skills)
Distributed tracing in serverless / Lambda (use aws-lambda-tracing — has cold-start propagation gotchas)
