instrument-a-distributed-trace

name	instrument-a-distributed-trace
category	code
description	Instrument distributed tracing with stable propagation, meaningful spans, bounded attributes, sampling, privacy, asynchronous links, and validation. Use when a request or workflow crosses services and queues.

instrument-a-distributed-trace

Trace decisions and boundaries without turning telemetry into a data leak.

When to use

• Use across HTTP, RPC, database, queue, job, and external-service boundaries.
• Never attach secrets, raw payloads, personal data, or unbounded identifiers.

Preconditions

• Define journeys, trust boundaries, standards, telemetry backend, retention, cost, privacy, and debugging needs.

Procedure

1. Choose trace and baggage propagation standards and explicit trust-boundary filtering.
2. Name spans by stable operation, not high-cardinality route values.
3. Record timing, status, bounded errors, dependency, version, region, and safe domain outcome.
4. Model synchronous parent-child and asynchronous producer-consumer links accurately.
5. Define head, tail, error, and rare-path sampling with known blind spots.
6. Correlate logs and metrics through safe identifiers.
7. Test propagation, broken context, retries, fan-out, long jobs, and sampling.
8. Measure overhead, cardinality, storage, access, and deletion.

Failure plan

• Drop unsafe attributes and continue bounded tracing rather than exporting sensitive data.

Worked example

A checkout trace links an asynchronous fraud result as a span link, preserving retries without falsely nesting hours of queue delay.

Done

• A tracing implementation records propagation, span model, attributes, async links, sampling, privacy, retention, and dashboards
• Cross-service, queue, retry, fan-out, sampling, cardinality, overhead, and sensitive-data tests verify telemetry