instrument-a-distributed-trace

# instrument-a-distributed-trace Trace decisions and boundaries without turning telemetry into a data leak. ## When to use - Use across HTTP, RPC, database, queue, job, and external-service boundaries. - Never attach secrets, raw payloads, personal data, or unbounded identifiers. ## Preconditions - Define journeys, trust boundaries, standards, telemetry backend, retention, cost, privacy, and debugging needs. ## Procedure 1. Choose trace and baggage propagation standards and explicit trust-boundary filtering. 2. Name spans by stable operation, not high-cardinality route values. 3. Record timing, status, bounded errors, dependency, version, region, and safe domain outcome. 4. Model synchronous parent-child and asynchronous producer-consumer links accurately. 5. Define head, tail, error, and rare-path sampling with known blind spots. 6. Correlate logs and metrics through safe identifiers. 7. Test propagation, broken context, retries, fan-out, long jobs, and sampling. 8. Measure overhead, cardinality, storage, access, and deletion. ## Failure plan - Drop unsafe attributes and continue bounded tracing rather than exporting sensitive data. ## Worked example A checkout trace links an asynchronous fraud result as a span link, preserving retries without falsely nesting hours of queue delay. ## Done - A tracing implementation records propagation, span model, attributes, async links, sampling, privacy, retention, and dashboards - Cross-service, queue, retry, fan-out, sampling, cardinality, overhead, and sensitive-data tests verify telemetry