otel-ottl

name

otel-ottl

description

OpenTelemetry Transformation Language (OTTL) expert for writing and debugging telemetry transformations in the OpenTelemetry Collector. Use when authoring or reviewing `transform`, `filter`, `tail_sampling` processor configs or `routing` connector configs, debugging OTTL syntax or semantics, transforming traces, metrics, logs, or profiles, or converting data-processing requirements into OTTL statements.

OpenTelemetry Transformation Language (OTTL)

OTTL is a domain-specific language for transforming telemetry inside the OpenTelemetry Collector. It is consumed by the transform, filter, and tail_sampling processors, the routing connector, and a few other components in opentelemetry-collector-contrib.

This skill targets pkg/ottl as of collector-contrib v0.156.0. Function and path availability differs across Collector releases; check the upstream pkg/ottl/ottlfuncs/README.md and pkg/ottl/contexts/*/README.md for the exact set in older or newer releases.

Statement syntax

function(arguments) [where condition]

Every statement has exactly one editor (lowercase: set, delete_key, append, …) optionally guarded by a where clause whose body is a boolean expression. Conditions can call converters (uppercase: Concat, IsMatch, ParseJSON, …) which return values but do not mutate telemetry.

set(span.attributes["env"], "prod") where resource.attributes["env"] == nil

Workflow

1. Pick the component. transform rewrites; filter drops; the routing connector fans out by pipeline; tail_sampling keeps/drops traces. The component decides which contexts and function set are usable.
2. Pick the context. resource, scope, span, spanevent, metric, datapoint, exemplar, log, profile, profilesample, or otelcol for Collector request/client metadata. Operate at the lowest level that gives you the data — using datapoint to set attributes is much cheaper than walking through metric.data_points from the metric context.
3. Write statements. Reach for references/quick-reference.md for common recipes; references/contexts.md for paths/enums; references/functions.md for the editor and converter catalog.
4. Set error_mode. ignore keeps the pipeline running and logs errors; silent does the same but quietly; propagate aborts on first failure. In v0.156, transform and filter default to ignore through enabled beta feature gates; routing defaults to propagate unless connector.routing.defaultErrorModeIgnore is enabled; lower-level OTTL sequences default to propagate.
5. Verify. OTTL gotchas are the kind that pass the eye test (see Common gotchas). Use the telemetrygen verification recipe — otelcol-contrib + file exporter + telemetrygen — to confirm the snippet does what the prose claims before shipping.

Contexts at a glance

OTTL paths are scoped by signal. Higher levels are reachable from lower ones (e.g., resource.attributes from a span statement); the reverse is not true.

Context	Common paths
Resource	resource.attributes["service.name"], resource.schema_url
Scope	scope.name, scope.version, scope.attributes["…"]
Span	span.name, span.kind, span.status.code, span.attributes["…"], span.flags
Span Event	spanevent.name, spanevent.attributes["…"], spanevent.event_index
Metric	metric.name, metric.unit, metric.type, metric.aggregation_temporality
DataPoint	datapoint.value_double, datapoint.value_int, datapoint.attributes["…"]
Exemplar	exemplar.double_value, exemplar.int_value, exemplar.filtered_attributes["…"] (transform metric_statements only, v0.156+)
Log	log.body, log.body.string, log.severity_number, log.attributes["…"]
Profile	profile.profile_id, profile.attributes["…"] (Development)
OTelCol	otelcol.client.metadata["x-tenant"][0], otelcol.grpc.metadata["x-tenant"][0] (read-only, enabled by default feature gate)

Full path inventory plus enums in references/contexts.md.

Essential functions

# Editors (mutate telemetry)
set(target, value)
delete_key(target, key)               # delete by exact key
delete_matching_keys(target, regex)   # delete by regex
delete_index(target, start, end?)     # remove one slice item or range (v0.145+)
keep_keys(target, [k1, k2])           # keep only these keys
merge_maps(target, source, "upsert")  # "insert" | "update" | "upsert"
truncate_all(target, max_len)         # UTF-8 safe by default in v0.148+
replace_pattern(target, regex, replacement)
stringify_all(target)                 # map values -> strings (v0.155+)

# Converters (return values)
Concat([a, b], "-")
Split(s, ",")
ToLowerCase(s) / ToUpperCase(s)
IsMatch(s, "pattern")                 # bool
String(v) / Int(v) / Double(v) / Bool(v)
ParseJSON(s) / ParseKeyValue(s, "=", "&")
URL(s)                                # parse URL components (v0.127+)
ExtractPatterns(s, "(?P<name>…)")     # named captures → map
ExtractGrokPatterns(s, "%{IP:client}") # Grok (v0.130+)
IsInCIDR(ip, ["10.0.0.0/8"])          # CIDR membership (v0.146+)
SHA256(s) / Murmur3Hash(s) / XXH3(s)  # hexadecimal string hashes
UUID() / UUIDv7()

Converter results can be indexed by dynamic string or integer expressions in v0.155+:

set(log.attributes["last"], Split(log.body.string, ",")[Len(Split(log.body.string, ",")) - 1])
set(log.attributes["selected"], ParseJSON(log.body.string)[log.attributes["field_name"]])

Full catalog with signatures in references/functions.md.

The transform processor also provides 17 signal-specific functions that are not part of the common pkg/ottl/ottlfuncs catalog. See the transform-only function table for their contexts, signatures, and released-source links.

Common patterns

# Conditional set
set(span.attributes["sampled"], true)
    where (span.end_time_unix_nano - span.start_time_unix_nano) > 1000000000

# Normalize a value
set(span.attributes["http.method"],
    ToUpperCase(String(span.attributes["http.method"])))

# Parse a JSON log body into structured attributes
set(log.attributes, ParseJSON(log.body.string))
    where IsString(log.body) and IsMatch(log.body.string, "^\\s*\\{.*\\}\\s*$")

# Mark errored HTTP spans
set(span.status.code, STATUS_CODE_ERROR)
    where IsInt(span.attributes["http.status_code"])
      and Int(span.attributes["http.status_code"]) >= 400

# Redact secrets by key pattern
delete_matching_keys(span.attributes, "(?i).*(password|secret|token|apikey).*")

# Hash PII rather than dropping it (preserves cardinality for analytics)
set(span.attributes["user.email_hash"], SHA256(span.attributes["user.email"]))
delete_key(span.attributes, "user.email")

More recipes (sampling, redaction, time math, parsing) in references/quick-reference.md.

Processor wiring

processors:
  transform:
    error_mode: ignore        # ignore | silent | propagate
    trace_statements:
      - context: span
        statements:
          - set(span.attributes["processed"], true)
    log_statements:
      - context: log
        statements:
          - set(log.attributes["source"], "collector")
    metric_statements:
      - context: datapoint
        statements:
          - set(datapoint.attributes["env"], "prod")

  filter:
    error_mode: ignore
    trace_conditions:
      - 'IsMatch(span.name, "^/health.*")'
    log_conditions:
      - 'log.severity_number < SEVERITY_NUMBER_WARN'

  tail_sampling:
    policies:
      - name: errors
        type: ottl_condition
        ottl_condition:
          span:
            - 'span.status.code == STATUS_CODE_ERROR'

connectors:
  routing:
    error_mode: ignore
    default_pipelines: [traces/default]
    table:
      - condition: 'resource.attributes["env"] == "prod"'
        pipelines: [traces/prod]
      - condition: 'span.status.code == STATUS_CODE_ERROR'
        pipelines: [traces/errors]

Common gotchas

These mistakes pass YAML validation but break OTTL semantics. Most have cost real time in production rollouts.

replace_pattern backreferences need $${1} in YAML

OTTL uses ${1}, ${2}, … for regex backreferences. The collector's YAML loader treats $ as an env-var marker, so YAML $$ becomes OTTL $. To produce ${1} at the OTTL level, write $${1} in YAML. Writing "$1REDACTED" produces literal $1REDACTED with no replacement — silent failure.

Go RE2 has a repeat-count ceiling

Patterns like (.{1024}).* fail to compile with "invalid repeat count". For length-based truncation, prefer Substring + Len:

set(attributes["db.statement"], Substring(attributes["db.statement"], 0, 1024))
    where attributes["db.statement"] != nil and Len(attributes["db.statement"]) > 1024

Or use truncate_all for whole maps (UTF-8 safe by default since v0.148):

truncate_all(span.attributes, 1024)

attributes processor ≠ resource processor

The OTel attributes processor only operates on span/log/metric attributes. To touch a resource attribute use the resource processor or a transform processor with context: resource. A config like attributes/strip_resource: actions: [...delete os.description...] runs without error but doesn't change resource attributes — silent no-op.

logdedup paths use dot-notation only

The processor accepts include_fields / exclude_fields (not fields). Paths must start with attributes. or body. and use dot-notation. Bracket notation (attributes["service.name"]) is rejected, and resource[...] paths are not addressable. Default behavior dedups on the full record, which is usually what's wanted.

k8sattributes cannot extract k8s.cluster.name

The processor's metadata list is restricted to pod-level identity. Cluster name has to come from resourcedetection or a static resource processor that reads it from an env var.

Path syntax changed in v0.120

In older configs you may see span_event.* — current syntax is spanevent.*. Cache paths now require the context prefix: write span.cache["x"] not just cache["x"]. Plain cache is only valid in profile/profilesample contexts where it's the documented path. Paste-from-old-config is the most common source of regressions.

Base64Decode is deprecated

Use Decode(value, "base64") instead. The same Decode converter handles base64-raw, base64-url, base64-raw-url, and IANA character set encodings. Keep Base64Decode only if pinned to a pre-v0.141 collector.

routing request context is deprecated

In routing connector configs, use otelcol.client.metadata["key"][0] for HTTP/client metadata or otelcol.grpc.metadata["key"][0] for gRPC metadata. The old context: request plus request["key"] == "value" form is deprecated in v0.156.

Bool converter coercion is loose

Bool preserves booleans, treats any non-zero integer or double as true, and uses Go boolean parsing for strings (1, t, T, TRUE, true, True, and their false equivalents). Invalid strings and unsupported types error; nil returns nil. Don't assume Python-like truthiness for arbitrary strings.

Verify before publishing

YAML/OTTL gotchas like the above pass the eye test. Use the telemetrygen verification recipe (otelcol-contrib + file exporter + telemetrygen) to confirm the snippet does what the surrounding prose claims, especially before shipping to a customer or production.

Best practices

1. Validate inputs before conversion: where IsString(x), where x != nil.
2. Order conditions by selectivity (cheap and most-selective first). span.kind == SPAN_KIND_SERVER and IsMatch(...) lets the kind check short-circuit before the regex runs.
3. Cache expensive operations in <context>.cache: set(span.cache["url_parts"], Split(span.attributes["http.url"], "/")), then read from cache.
4. Use the most specific context. datapoint for metric-attribute work beats walking metric.data_points from the metric context.
5. Escape regex correctly. Use \\d+, \\., \\s+ in OTTL strings (single backslash in YAML becomes a literal).
6. Prefer keep_keys to a long list of delete_key when shaping output — easier to read, fails closed.

Versioning notes

Recently added (still useful to know which release introduced them when supporting users on older collectors):

Feature	Since
Exemplar context (transform metric_statements only)	v0.156
Routing connector context inference (condition with context-qualified paths)	v0.156
Routing connector request context deprecated; use otelcol.* metadata paths	v0.156
connector.routing.defaultErrorModeIgnore feature gate	v0.155
otelcol.* context via enabled-by-default ottl.contexts.enableOTelColContext feature gate	v0.147
Filter processor top-level trace_conditions, metric_conditions, log_conditions	v0.146
Profile / ProfileSample contexts	v0.124 / v0.132 (Development)
Cache paths require context prefix; spanevent rename	v0.120 (breaking)
delete_index editor	v0.145
span.flags path	v0.145
truncate_all UTF-8 safe default (utf8_safe parameter)	v0.148 (behavior change)
Substring optional utf8_safe parameter (default false)	v0.156
SpanID / TraceID accept hex strings	v0.142
flatten resolveConflicts parameter	v0.139
Base64Encode	v0.147
Decode, deprecates Base64Decode	v0.141
Bool converter	v0.143
ExtractGrokPatterns	v0.130
URL, UserAgent	v0.127, v0.134
IsInCIDR	v0.146
Murmur3Hash*, XXH3, XXH128	v0.129, v0.135
Sort, Index, SliceToMap	v0.125, v0.126, v0.128
UUIDv7, ParseSeverity, CommunityID	v0.138, v0.133, v0.131
stringify_all editor; dynamic keys when indexing converter results	v0.155

References

• references/contexts.md — context paths and enums
• references/functions.md — editors and converters with signatures
• references/quick-reference.md — recipes, regex patterns, troubleshooting
• Upstream: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/pkg/ottl