telemetry-foundations

# Telemetry Foundations ## Overview A shared vocabulary for observability across HVE Core agents. This skill describes *what* telemetry data exists and *how it is named*, not which SDK or vendor to use. Agents producing planning artifacts (ADRs, PRDs, security/RAI plans, code-review reports) and agents producing user-facing application code reference this skill so that downstream pipelines (traces, metrics, logs) speak a consistent OpenTelemetry-aligned language. ## When to Apply Apply this skill in the following situations: * Any agent producing user-facing application code that emits spans, metrics, or structured logs. * Architecture Decision Records that touch observability, monitoring, or audit logging. * Code-review reports that flag telemetry gaps, inconsistent span naming, or unbounded metric cardinality. * Security or Responsible AI plans that cite audit logs, traceability, or evidence chains. * Product or business requirement documents that specify success metrics expressed as service telemetry. ## Core Principles The vocabulary in this skill follows five principles: * Declarative, not prescriptive. Define the names and shapes; leave the choice of SDK, exporter, and backend to the implementing team. * OpenTelemetry-aligned. Trace, metric, and log models follow the OTel data model so artifacts remain portable. * Semantic conventions first. Where an OTel semantic convention exists for a domain (HTTP, RPC, database, messaging, GenAI, FaaS), prefer it over a bespoke attribute. * PII by denylist. Treat PII as default-deny via the denylist in [references/pii-denylist.md](references/pii-denylist.md); any field listed there requires an explicit redaction strategy before it can be emitted. * Vendor-agnostic. Avoid coupling vocabulary to a single backend; OTLP is the assumed wire protocol. ## Trace Vocabulary Spans describe a unit of work and its causal relationship to other work. Span kinds: * `server` - inbound request handled by this service. * `client` - outbound request issued by this service. * `producer` - asynchronous message published to a queue or topic. * `consumer` - asynchronous message received from a queue or topic. * `internal` - in-process operation with no remote peer. Required resource-scoped attributes on every span: * `service.name` * `service.version` * `deployment.environment` Span naming pattern: `<verb>.<resource>` using lowercase dot-separated tokens. The verb describes the operation (`get`, `create`, `publish`, `consume`, `query`); the resource describes the target entity (`order`, `customer`, `payment.intent`). Examples: `get.order`, `publish.order.created`, `query.customer.by_email`. For domains covered by OTel semantic conventions (HTTP, RPC, database, messaging, GenAI, FaaS), use the convention's span-naming guidance instead of the generic pattern above. ## Metric Vocabulary Metrics describe aggregate measurements over time. Instrument types: * `counter` - monotonic, additive (request count, bytes sent). * `up-down-counter` - non-monotonic, additive (queue depth, active connections). * `histogram` - distribution of values (request duration, payload size). * `gauge` - last-sampled value, non-additive (CPU temperature, memory in use). * `observable-counter`, `observable-up-down-counter`, `observable-gauge` - async variants polled by the SDK. Unit conventions follow [UCUM](https://ucum.org/ucum). Examples: `s` (seconds), `ms` (milliseconds), `By` (bytes), `1` (dimensionless count). Express durations as histograms in seconds (`s`) by default to align with OTel HTTP semantic conventions. Metric naming pattern: `<domain>.<entity>.<measure>` using lowercase dot-separated tokens. Examples: `http.server.request.duration`, `db.client.connections.usage`, `messaging.publish.duration`. Cardinality discipline: every attribute attached to a metric multiplies the time-series count. Bound high-cardinality dimensions (user ID, request ID, free-form strings) at the source or move them to exemplars and traces. ## Log Vocabulary Structured logs carry discrete events with severity and context. Severity levels (OTel log data model): * `TRACE` - fine-grained diagnostic detail, off by default. * `DEBUG` - diagnostic detail useful during development. * `INFO` - normal operational events. * `WARN` - unexpected condition that does not block the operation. * `ERROR` - operation failed; the caller likely saw a failure. * `FATAL` - process is going to terminate. Recommended structured fields on every log record: * `timestamp` (ISO-8601, UTC). * `severity_text` and `severity_number`. * `body` (the human-readable message; structured data goes in `attributes`). * `attributes.*` (typed key-value pairs scoped to this event). * `resource.*` (inherited from the producing service). Trace correlation: when a log record is emitted within an active span, inject `trace_id` and `span_id` so traces and logs join cleanly downstream. Logging libraries that integrate with the OTel context propagator do this automatically. ## PII Handling Personally Identifiable Information is handled by denylist. The authoritative list lives in [references/pii-denylist.md](references/pii-denylist.md). Treat any field on that list as default-deny: do not emit it as a span attribute, metric dimension, or log field without an explicit redaction strategy. Redaction patterns: * Hash - one-way hash (SHA-256, optionally truncated) for fields that must remain joinable across events but should not be human-readable. * Drop - omit the field entirely from telemetry. * Tokenize - replace with an opaque token resolvable only through a separate, access-controlled store. Identifier convention: where a stable user reference is needed in telemetry, use `user.id` populated with an opaque hash of the canonical user identifier, never the raw email, phone, or external account ID. When introducing a new attribute that could contain PII, add it to the denylist first and choose a redaction strategy before emitting it. ## Sampling and Cost Sampling controls the volume of telemetry shipped to downstream collectors. Sampling strategies: * Head-based - decision made at span start, propagated through the trace. Low cost, simple, but cannot bias toward late-discovered properties (such as errors). * Tail-based - decision made after a trace completes, typically in a collector. Higher cost, allows policies such as "keep all error traces" and "keep slow traces". Defaults: * Use a parent-based sampler so child spans inherit the parent's sampling decision and traces remain whole. * When tail-based sampling is available, bias toward keeping error traces and a representative sample of successful traces. Metric and log sampling: metrics are pre-aggregated and rarely sampled; logs are typically rate-limited per severity rather than sampled. ## Resource Attributes Resource attributes describe the entity producing the telemetry and are attached to every span, metric, and log record automatically by the SDK. Required: * `service.name` * `service.version` * `deployment.environment` * `telemetry.sdk.name` * `telemetry.sdk.language` * `telemetry.sdk.version` Recommended when applicable: * `cloud.*` (cloud provider, region, account ID). * `k8s.*` (cluster name, namespace, pod name). * `host.*` (hostname, architecture, OS type). Follow the OTel [Resource Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/resource/) for canonical attribute names. ## Decision Tree Use this quick-select when choosing whether and how to instrument: 1. Is this user-facing or part of a user-visible flow? If no, prefer DEBUG logs and skip span/metric emission unless needed for capacity planning. 2. Is the cardinality of the proposed attributes bounded? If no, move the unbounded field to a log attribute or trace exemplar rather than a metric dimension. 3. Does the data contain or derive from a field in [references/pii-denylist.md](references/pii-denylist.md)? If yes, apply a redaction strategy before emitting. 4. Does the operation cross a service boundary (network, queue, process)? If yes, emit a span with the matching `server`, `client`, `producer`, or `consumer` kind and propagate context. 5. Is the operation high-volume? If yes, rely on parent-based sampling and (where available) tail-based policies; do not disable instrumentation outright. 6. Does an OpenTelemetry semantic convention cover this domain? If yes, use its attribute names and span-naming guidance; if no, follow the naming patterns in this skill and propose a new entry in [references/proposed-additions.md](references/proposed-additions.md). ## References Authoritative external sources: * [OpenTelemetry Semantic Conventions v1.41.0](https://opentelemetry.io/docs/specs/semconv/) * [W3C Trace Context](https://www.w3.org/TR/trace-context/) * [OpenTelemetry Protocol (OTLP) Specification](https://opentelemetry.io/docs/specs/otlp/) * [OpenTelemetry Logs Data Model](https://opentelemetry.io/docs/specs/otel/logs/data-model/) * [OpenTelemetry FaaS Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/faas/) Portions adapted from OpenTelemetry Semantic Conventions, (C) OpenTelemetry Authors, licensed under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/). Internal: * [references/pii-denylist.md](references/pii-denylist.md) - authoritative PII denylist with redaction strategies. * [references/proposed-additions.md](references/proposed-additions.md) - intake for new vocabulary proposals.