opentelemetry

name	opentelemetry
description	Instrument applications and infrastructure with OpenTelemetry for unified traces, metrics, and logs. Use when implementing distributed tracing, service-level troubleshooting, or vendor-neutral observability.
license	MIT
metadata	{"author":"devops-skills","version":"1.0"}

OpenTelemetry

Adopt vendor-neutral telemetry with consistent instrumentation across services.

When to Use This Skill

• Debugging latency across microservices
• Standardizing observability data model and naming
• Sending telemetry to Prometheus, Grafana, Datadog, or OTLP backends
• Building SLO dashboards with trace-to-log correlation
• Instrumenting Python or Node.js applications with tracing and metrics
• Setting up auto-instrumentation for existing services without code changes

Prerequisites

• Application services running in containers or on VMs
• Backend for traces (Jaeger, Tempo, Datadog, or any OTLP receiver)
• Backend for metrics (Prometheus, Mimir, or OTLP receiver)
• Kubernetes cluster (for collector deployment) or VM with systemd
• Network access from services to collector, and collector to backends

Core Workflow

1. Define semantic conventions for services, environments, and versions.
2. Add SDK or auto-instrumentation in each service.
3. Run an OpenTelemetry Collector to receive, transform, and export telemetry.
4. Validate cardinality and sampling to control cost.
5. Create golden signals dashboards and alerting from collected data.

Collector Production Configuration

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

  # Scrape Prometheus endpoints
  prometheus:
    config:
      scrape_configs:
        - job_name: "kubernetes-pods"
          kubernetes_sd_configs:
            - role: pod
          relabel_configs:
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
              action: keep
              regex: "true"
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
              action: replace
              target_label: __address__
              regex: (.+)
              replacement: $$1

  # Host metrics for infrastructure monitoring
  hostmetrics:
    collection_interval: 30s
    scrapers:
       {}
       {}
       {}
       {}
       {}


  
     
     

  
     
     
     

  
    
        
         
         

  
  
    
      
         

  
  
    
        
        
             
             

  
  
     [, , , , ]
     


  
  
     
    
       

  
  
     
    
       

  
  
     
    
       

  
  
     


  
    
       
    
       

  
    
       []
       [, , , , ]
       []
    
       [, , ]
       [, , , ]
       []
    
       []
       [, , , ]
       []

Collector Kubernetes Deployment

# otel-collector-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector
  namespace: observability
spec:
  replicas: 2
  selector:
    matchLabels:
      app: otel-collector
  template:
    metadata:
      labels:
        app: otel-collector
    spec:
      containers:
        - name: collector
          image: otel/opentelemetry-collector-contrib:0.98.0
          args: ["--config=/etc/otel/config.yaml"]
          ports:
            - containerPort: 4317
              name: otlp-grpc
            - containerPort: 4318
              name: otlp-http
            - containerPort: 8888
              name: metrics
          resources:
            requests:
              cpu: 200m
              memory: 256Mi
            
               
               
          
              
               
          
            
               
               
          
            
               
               
      
          
          
             

 
 

   
   

  
     
  
      
       
       
      
       
       
      
       
       

Python SDK Instrumentation

# tracing_setup.py
"""Initialize OpenTelemetry tracing and metrics for a Python service."""
from opentelemetry import trace, metrics
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.sdk.resources import Resource
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.sqlalchemy import SQLAlchemyInstrumentor
import os

def init_telemetry(service_name: str, service_version: str):
    """Initialize OTel SDK with traces and metrics."""
    resource = Resource.create({
        "service.name": service_name,
        "service.version": service_version,
        "deployment.environment": os.getenv("DEPLOY_ENV", "development"),
    })

    # Traces
    trace_exporter = OTLPSpanExporter(
        endpoint=os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://otel-collector:4317"),
        insecure=,
    )
    tracer_provider = TracerProvider(resource=resource)
    tracer_provider.add_span_processor(BatchSpanProcessor(trace_exporter))
    trace.set_tracer_provider(tracer_provider)

    
    metric_exporter = OTLPMetricExporter(
        endpoint=os.getenv(, ),
        insecure=,
    )
    metric_reader = PeriodicExportingMetricReader(metric_exporter, export_interval_millis=)
    meter_provider = MeterProvider(resource=resource, metric_readers=[metric_reader])
    metrics.set_meter_provider(meter_provider)

    
    RequestsInstrumentor().instrument()
    SQLAlchemyInstrumentor().instrument()

     trace.get_tracer(service_name), metrics.get_meter(service_name)


tracer, meter = init_telemetry(, )


 tracer.start_as_current_span()  span:
    span.set_attribute(, order_id)
    span.set_attribute(, total)
    


request_counter = meter.create_counter(
    ,
    description=,
)
request_counter.add(, {: , : })

Node.js SDK Instrumentation

// tracing.js
// Initialize OpenTelemetry for a Node.js service.
// Load this file BEFORE any other imports: node -r ./tracing.js app.js
const { NodeSDK } = require("@opentelemetry/sdk-node");
const { OTLPTraceExporter } = require("@opentelemetry/exporter-trace-otlp-grpc");
const { OTLPMetricExporter } = require("@opentelemetry/exporter-metrics-otlp-grpc");
const { PeriodicExportingMetricReader } = require("@opentelemetry/sdk-metrics");
const { getNodeAutoInstrumentations } = require("@opentelemetry/auto-instrumentations-node");
const { Resource } = require("@opentelemetry/resources");
const { ATTR_SERVICE_NAME, ATTR_SERVICE_VERSION } = require("@opentelemetry/semantic-conventions");

const resource = new Resource({
  [ATTR_SERVICE_NAME]: process.env.SERVICE_NAME || "node-service",
  [ATTR_SERVICE_VERSION]: process.env.SERVICE_VERSION || "1.0.0",
  "deployment.environment": process.env.DEPLOY_ENV || ,
});

 sdk =  ({
  resource,
  :  ({
    : process.. || ,
  }),
  :  ({
    :  ({
      : process.. || ,
    }),
    : ,
  }),
  : [
    ({
      : {
        : [, ],
      },
      : { :  },
      : { :  },
      : { :  },
    }),
  ],
});

sdk.();
process.(,  sdk.());

Auto-Instrumentation with Kubernetes Operator

# otel-auto-instrumentation.yaml
# Install the OTel Operator first:
#   helm install opentelemetry-operator open-telemetry/opentelemetry-operator \
#     --namespace observability --create-namespace

# Define instrumentation for Python services
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: python-instrumentation
  namespace: default
spec:
  exporter:
    endpoint: http://otel-collector.observability:4317
  propagators:
    - tracecontext
    - baggage
  sampler:
    type: parentbased_traceidratio
    argument: "0.25"
  python:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:0.44b0
    env:
      - name: OTEL_PYTHON_LOG_CORRELATION
        value: "true"
---
# Define instrumentation for Node.js services
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: nodejs-instrumentation
  namespace: default
spec:
  
     
  
     
     
  
     
     
  
     

To instrument a pod, add the annotation:

# For Python:
metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-python: "true"

# For Node.js:
metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-nodejs: "true"

Sampling Strategies

# Tail-based sampling config (in collector)
processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 100000
    policies:
      # Always keep error traces
      - name: errors
        type: status_code
        status_code:
          status_codes: [ERROR]

      # Always keep slow traces (> 2s)
      - name: slow-traces
        type: latency
        latency:
          threshold_ms: 2000

      # Sample 10% of successful traces
      - name: normal-traffic
        type: probabilistic
        probabilistic:
          sampling_percentage: 10

      # Always keep traces with specific attributes
      - name: important-users
        type: string_attribute
        string_attribute:
          key: user.tier
          values: [enterprise, premium]

      # Rate limit per service to prevent one service from dominating
      - name: 
         
        
           

Best Practices

• Use tail-based sampling for high-volume production traces.
• Tag telemetry with service.name, service.version, and deployment.environment.
• Drop noisy attributes early in the collector.
• Keep metric label cardinality low for stable query performance.
• Use resource detectors to automatically populate cloud metadata.
• Separate collector pools for traces vs metrics if volume requires it.
• Set memory_limiter on every collector pipeline to prevent OOM.
• Use the contrib collector image for production (includes more receivers/exporters).

Troubleshooting

Symptom	Check	Fix
No traces arriving at backend	Collector logs for export errors	Verify endpoint URL and network policy
Missing spans in a trace	Propagation headers stripped by proxy	Configure proxy to pass traceparent header
High memory on collector	Too many in-flight traces for tail sampling	Reduce num_traces or increase memory limit
Metric cardinality explosion	Unbounded label values (user IDs, URLs)	Add transform processor to normalize values
Auto-instrumentation not working	Pod annotation missing or operator not running	Verify operator is healthy and annotation is correct
Duplicate metrics	Both SDK and auto-instrumentation active	Use only one instrumentation method per signal

Related Skills

• prometheus-grafana - Dashboarding and alerting
• datadog - Managed observability backend
• alerting-oncall - On-call routing and escalation
• rag-observability-evals - RAG-specific observability
• agent-observability - AI agent tracing