dapr-observability-setup

name	dapr-observability-setup
description	Configure OpenTelemetry tracing, metrics, and structured logging for DAPR applications. Integrates with Azure Monitor, Jaeger, Prometheus, and other observability backends.

DAPR Observability Setup

Configure comprehensive observability for DAPR microservices.

When to Activate

This skill should be invoked when:

• Setting up a new DAPR project
• Adding monitoring/tracing to existing services
• Configuring Azure Monitor or other backends
• Debugging distributed system issues

Observability Components

1. Distributed Tracing (OpenTelemetry)

Configure tracing in Python applications:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor

def configure_tracing(service_name: str):
    provider = TracerProvider()
    processor = BatchSpanProcessor(OTLPSpanExporter())
    provider.add_span_processor(processor)
    trace.set_tracer_provider(provider)

    # Instrument FastAPI
    FastAPIInstrumentor.instrument()

    # Instrument HTTP requests
    RequestsInstrumentor().instrument()

2. Metrics (Prometheus)

Configure metrics collection:

from prometheus_client import Counter, Histogram, start_http_server

# Define metrics
REQUEST_COUNT = Counter(
    'dapr_requests_total',
    'Total requests',
    ['method', 'endpoint', 'status']
)

REQUEST_LATENCY = Histogram(
    'dapr_request_duration_seconds',
    'Request latency',
    ['method', 'endpoint']
)

# Start metrics server
start_http_server(9090)

3. Structured Logging

Configure JSON logging:

import logging
import json
from datetime import datetime

class JSONFormatter(logging.Formatter):
    def format(self, record):
        log_obj = {
            "timestamp": datetime.utcnow().isoformat(),
            "level": record.levelname,
            "message": record.getMessage(),
            "service": "my-service",
            "trace_id": getattr(record, 'trace_id', None),
            "span_id": getattr(record, 'span_id', None),
        }
        return json.dumps(log_obj)

def configure_logging():
    handler = logging.StreamHandler()
    handler.setFormatter(JSONFormatter())
    logging.root.handlers = [handler]
    logging.root.setLevel(logging.INFO)

DAPR Configuration

Tracing Configuration

apiVersion: dapr.io/v1alpha1
kind: Configuration
metadata:
  name: tracing-config
spec:
  tracing:
    samplingRate: "1"  # 100% for dev, reduce for production
    otel:
      endpointAddress: "otel-collector:4317"
      isSecure: false
      protocol: grpc

Azure Monitor Integration

apiVersion: dapr.io/v1alpha1
kind: Configuration
metadata:
  name: azure-monitor-config
spec:
  tracing:
    samplingRate: "0.1"  # 10% sampling for production
    otel:
      endpointAddress: "https://dc.services.visualstudio.com/v2/track"
      isSecure: true
      protocol: http

Observability Stack

OpenTelemetry Collector Config

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 1s
    send_batch_size: 1024

exporters:
  jaeger:
    endpoint: jaeger:14250
    tls:
      insecure: true

  prometheus:
    endpoint: 0.0.0.0:8889

  azuremonitor:
    connection_string: ${APPLICATIONINSIGHTS_CONNECTION_STRING}

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [jaeger, azuremonitor]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus]

Best Practices

1. Correlation IDs: Always propagate trace context
2. Sampling: Use 10-20% sampling in production
3. Log Levels: Use INFO for normal, DEBUG for troubleshooting
4. Metrics Cardinality: Limit label values to prevent explosion
5. Retention: Set appropriate retention for traces (7-30 days)