Skip to main content Skills Marketplace Discover and explore AI skills built by the community.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Show prompt details A direct command skips the review prompt. Inspect the source before running it.
npx skills add https://github.com/bobmatnyc/claude-mpm-skills --skill golang-observability-opentelemetryThe command stays on one line. Scroll horizontally to inspect it before copying.
Prefer a local copy? Download the files currently available to SkillsMP.
Download Zip Downloading... name golang-observability-opentelemetry description Instrumenting Go applications with OpenTelemetry for distributed tracing, Prometheus for metrics, and structured logging with slog user-invocable false disable-model-invocation true version 1.0.0 category toolchain author Claude MPM Team license MIT progressive_disclosure {"entry_point":{"summary":"Master Go observability through OpenTelemetry distributed tracing, Prometheus metrics, and slog structured logging for production-ready microservices","when_to_use":"Instrumenting microservices for observability, setting up distributed tracing across services, creating operational dashboards, debugging production issues, performance monitoring and optimization","quick_start":"1. Initialize OpenTelemetry tracer and exporter 2. Add Prometheus metrics endpoint 3. Configure slog for structured logging 4. Instrument HTTP middleware 5. Implement health checks"},"token_estimate":{"entry":150,"full":5000}} context_limit 700 tags ["observability","golang","opentelemetry","prometheus","tracing","metrics","logging","slog"] requires_tools []
Go Observability with OpenTelemetry
Overview
Modern Go applications require comprehensive observability through the three pillars: traces, metrics, and logs. OpenTelemetry provides vendor-neutral instrumentation for distributed tracing, Prometheus offers powerful metrics collection, and Go's slog package (1.21+) delivers structured logging with minimal overhead.
Key Features:
🔍 OpenTelemetry : Distributed tracing with context propagation
📊 Prometheus : Metrics collection with /metrics endpoint
📝 Structured Logging : slog with JSON formatting and correlation IDs
🎯 Auto-Instrumentation : HTTP/gRPC middleware patterns
💚 Health Checks : Kubernetes-ready readiness/liveness probes
🔄 Graceful Shutdown : Clean exporter shutdown and signal handling
When to Use This Skill
Activate this skill when:
Instrumenting microservices for production observability
Setting up distributed tracing across service boundaries
Creating operational dashboards with Prometheus/Grafana
Debugging production performance issues or bottlenecks
Implementing SLOs and monitoring SLIs
Adding observability to existing Go applications
Correlating logs, traces, and metrics for debugging
Core Observability Principles
The Three Pillars
Traces : Understand request flow across distributed systemsMetrics : Measure system behavior and performance over timeLogs : Record discrete events for debugging and audit
Correlation Strategy
All three pillars must share common identifiers:
Trace ID : Links all operations in a requestSpan ID : Identifies specific operation within traceRequest ID : Correlates logs with traces and metrics
OpenTelemetry Integration
Installation
go get go.opentelemetry.io/otel
go get go.opentelemetry.io/otel/sdk
go get go.opentelemetry.io/otel/exporters/jaeger
go get go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp
Basic Setup
package main
import (
"context"
"log"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/exporters/jaeger"
"go.opentelemetry.io/otel/sdk/resource"
"go.opentelemetry.io/otel/sdk/trace"
"go.opentelemetry.io/otel/semconv/v1.21.0"
func initTracer (serviceName string )
error
"http://localhost:14268/api/traces"
if
nil
return
nil
"1.0.0"
if
nil
return
nil
return
nil
func main ()
"order-service"
if
nil
defer
func ()
if
nil
"Error shutting down tracer: %v"
Creating Spans import (
"context"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/codes"
"go.opentelemetry.io/otel/trace"
)
func ProcessOrder (ctx context.Context, order Order) error {
tracer := otel.Tracer("order-service" )
ctx, span := tracer.Start(ctx, "ProcessOrder" )
defer span.End()
span.SetAttributes(
attribute.String("order.id" , order.ID),
attribute.Int("order.items" , len (order.Items)),
attribute.Float64("order.total" , order.Total),
)
if err := validateOrder(ctx, order); err != nil {
span.RecordError(err)
span.SetStatus(codes.Error, "validation failed" )
return err
}
if err := fulfillOrder(ctx, order); err != nil {
span.RecordError(err)
span.SetStatus(codes.Error, "fulfillment failed" )
return err
}
span.SetStatus(codes.Ok, "order processed successfully" )
return nil
}
func validateOrder (ctx context.Context, order Order) error {
_, span := otel.Tracer("order-service" ).Start(ctx, "validateOrder" )
defer span.End()
return nil
}
HTTP Middleware Instrumentation import (
"net/http"
"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)
func main () "order-handler" )
http.Handle("/orders" , wrappedHandler)
http.ListenAndServe(":8080" , nil )
}
func orderHandler (w http.ResponseWriter, r *http.Request) "order-service" )
ctx, span := tracer.Start(ctx, "orderHandler" )
defer span.End()
orderID := r.URL.Query().Get("id" )
span.SetAttributes(attribute.String("order.id" , orderID))
order, err := fetchOrder(ctx, orderID)
if err != nil {
span.RecordError(err)
http.Error(w, "Order not found" , http.StatusNotFound)
return
}
}
Prometheus Metrics
Installation go get github.com/prometheus/client_golang/prometheus
go get github.com/prometheus/client_golang/prometheus/promhttp
Metric Types and Patterns package metrics
import (
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promauto"
)
var (
httpRequestsTotal = promauto.NewCounterVec(
prometheus.CounterOpts{
Name: "http_requests_total" ,
Help: "Total number of HTTP requests" ,
},
[]string {"method" , "path" , "status" },
)
activeConnections = promauto.NewGauge(
prometheus.GaugeOpts{
Name: "active_connections" ,
Help: "Number of active connections" ,
},
)
httpRequestDuration = promauto.NewHistogramVec(
prometheus.HistogramOpts{
Name: "http_request_duration_seconds" ,
Help: "HTTP request duration in seconds" ,
Buckets: prometheus.DefBuckets,
},
[]string {"method" , "path" },
)
dbQueryDuration = promauto.NewSummaryVec(
prometheus.SummaryOpts{
Name: "db_query_duration_seconds" ,
Help: "Database query duration" ,
Objectives: map [float64 ]float64 {0.5 : 0.05 , 0.9 : 0.01 , 0.99 : 0.001 },
},
[]string {"query_type" },
)
)
Metrics Middleware import (
"net/http"
"strconv"
"time"
"github.com/prometheus/client_golang/prometheus/promhttp"
)
func MetricsMiddleware (next http.Handler) return http.HandlerFunc(func (w http.ResponseWriter, r *http.Request) defer activeConnections.Dec()
rw := &responseWriter{ResponseWriter: w, statusCode: http.StatusOK}
next.ServeHTTP(rw, r)
duration := time.Since(start).Seconds()
httpRequestDuration.WithLabelValues(r.Method, r.URL.Path).Observe(duration)
httpRequestsTotal.WithLabelValues(r.Method, r.URL.Path, strconv.Itoa(rw.statusCode)).Inc()
})
}
type responseWriter struct {
http.ResponseWriter
statusCode int
}
func (rw *responseWriter) int ) {
rw.statusCode = code
rw.ResponseWriter.WriteHeader(code)
}
func main () "/metrics" , promhttp.Handler())
handler := MetricsMiddleware(http.HandlerFunc(orderHandler))
http.Handle("/orders" , handler)
http.ListenAndServe(":8080" , nil )
}
Custom Metrics Example func ProcessPayment (ctx context.Context, payment Payment) error {
timer := prometheus.NewTimer(dbQueryDuration.WithLabelValues("payment_insert" ))
defer timer.ObserveDuration()
if err := db.InsertPayment(payment); err != nil {
httpRequestsTotal.WithLabelValues("POST" , "/payments" , "500" ).Inc()
return err
}
httpRequestsTotal.WithLabelValues("POST" , "/payments" , "200" ).Inc()
return nil
}
Structured Logging with slog
Basic Setup (Go 1.21+) package main
import (
"context"
"log/slog"
"os"
)
func initLogger () true ,
})
logger := slog.New(handler)
slog.SetDefault(logger)
return logger
}
func main () "service starting" ,
"service" , "order-service" ,
"version" , "1.0.0" ,
"port" , 8080 ,
)
}
Context-Aware Logging import (
"context"
"log/slog"
"go.opentelemetry.io/otel/trace"
)
func LoggerWithTrace (ctx context.Context) return slog.With(
"trace_id" , spanCtx.TraceID().String(),
"span_id" , spanCtx.SpanID().String(),
)
}
func HandleRequest (ctx context.Context, req Request) error {
logger := LoggerWithTrace(ctx)
logger.Info("processing request" ,
"request_id" , req.ID,
"method" , req.Method,
"path" , req.Path,
)
if err := processRequest(ctx, req); err != nil {
logger.Error("request failed" ,
"error" , err,
"duration_ms" , time.Since(req.StartTime).Milliseconds(),
)
return err
}
logger.Info("request completed successfully" ,
"duration_ms" , time.Since(req.StartTime).Milliseconds(),
)
return nil
}
Log Levels and Structured Fields func ProcessOrder (ctx context.Context, order Order) error {
logger := LoggerWithTrace(ctx).With(
"order_id" , order.ID,
"user_id" , order.UserID,
)
logger.Debug("validating order" , "items" , len (order.Items))
if len (order.Items) == 0 {
logger.Warn("empty order received" )
return ErrEmptyOrder
}
logger.Info("order validation passed" )
if err := fulfillOrder(ctx, order); err != nil {
logger.Error("fulfillment failed" ,
"error" , err,
slog.Group("order_details" ,
"total" , order.Total,
"items" , len (order.Items),
),
)
return err
}
logger.Info("order processed successfully" ,
"total" , order.Total,
)
return nil
}
Health Checks and Graceful Shutdown
Health Check Endpoints import (
"context"
"database/sql"
"encoding/json"
"net/http"
"time"
)
type HealthChecker struct {
db *sql.DB
}
type HealthStatus struct {
Status string `json:"status"`
Version string `json:"version"`
Checks map [string ]string `json:"checks"`
Timestamp time.Time `json:"timestamp"`
}
func (hc *HealthChecker) "Content-Type" , "application/json" )
w.WriteHeader(http.StatusOK)
json.NewEncoder(w).Encode(map [string ]string {
"status" : "alive" ,
})
}
func (hc *HealthChecker) 5 *time.Second)
defer cancel()
status := HealthStatus{
Status: "ready" ,
Version: "1.0.0" ,
Checks: make (map [string ]string ),
Timestamp: time.Now(),
}
if err := hc.db.PingContext(ctx); err != nil {
status.Status = "not_ready"
status.Checks["database" ] = "unhealthy: " + err.Error()
w.WriteHeader(http.StatusServiceUnavailable)
} else {
status.Checks["database" ] = "healthy"
}
w.Header().Set("Content-Type" , "application/json" )
if status.Status == "ready" {
w.WriteHeader(http.StatusOK)
}
json.NewEncoder(w).Encode(status)
}
Graceful Shutdown import (
"context"
"net/http"
"os"
"os/signal"
"syscall"
"time"
)
func main () "order-service" )
if err != nil {
log.Fatal(err)
}
server := &http.Server{
Addr: ":8080" ,
Handler: setupRoutes(),
}
shutdown := make (chan os.Signal, 1 )
signal.Notify(shutdown, os.Interrupt, syscall.SIGTERM)
go func () "server starting" , "port" , 8080 )
if err := server.ListenAndServe(); err != http.ErrServerClosed {
log.Fatal(err)
}
}()
<-shutdown
slog.Info("shutdown signal received" )
ctx, cancel := context.WithTimeout(context.Background(), 30 *time.Second)
defer cancel()
slog.Info("shutting down HTTP server" )
if err := server.Shutdown(ctx); err != nil {
slog.Error("HTTP server shutdown error" , "error" , err)
}
slog.Info("shutting down tracer" )
if err := tp.Shutdown(ctx); err != nil {
slog.Error("tracer shutdown error" , "error" , err)
}
slog.Info("shutdown complete" )
}
Complete Instrumentation Example package main
import (
"context"
"database/sql"
"log/slog"
"net/http"
"os"
"time"
"github.com/prometheus/client_golang/prometheus/promhttp"
"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/attribute"
)
type Server struct {
db *sql.DB
logger *slog.Logger
}
func (s *Server) "order-service" )
ctx, span := tracer.Start(ctx, "orderHandler" )
defer span.End()
logger := s.logger.With(
"trace_id" , span.SpanContext().TraceID().String(),
"request_id" , r.Header.Get("X-Request-ID" ),
)
orderID := r.URL.Query().Get("id" )
span.SetAttributes(attribute.String("order.id" , orderID))
logger.Info("fetching order" , "order_id" , orderID)
order, err := s.fetchOrder(ctx, orderID)
if err != nil {
span.RecordError(err)
logger.Error("failed to fetch order" , "error" , err)
http.Error(w, "Order not found" , http.StatusNotFound)
return
}
logger.Info("order fetched successfully" ,
"order_id" , orderID,
"items" , len (order.Items),
)
w.Header().Set("Content-Type" , "application/json" )
json.NewEncoder(w).Encode(order)
}
func (s *Server) string ) (*Order, error ) {
_, span := otel.Tracer("order-service" ).Start(ctx, "fetchOrder" )
defer span.End()
start := time.Now()
var order Order
err := s.db.QueryRowContext(ctx, "SELECT * FROM orders WHERE id = ?" , orderID).Scan(&order)
duration := time.Since(start).Seconds()
dbQueryDuration.WithLabelValues("select_order" ).Observe(duration)
return &order, err
}
func setupRoutes (s *Server, hc *HealthChecker) "/health" , hc.LivenessHandler)
mux.HandleFunc("/ready" , hc.ReadinessHandler)
mux.Handle("/metrics" , promhttp.Handler())
orderHandler := http.HandlerFunc(s.orderHandler)
mux.Handle("/orders" , otelhttp.NewHandler(orderHandler, "orders" ))
return MetricsMiddleware(mux)
}
Decision Trees
When to Use OpenTelemetry
Building distributed systems with multiple services
Need to trace requests across service boundaries
Debugging performance issues in microservices
Want vendor-neutral observability (switch backends easily)
Require correlation between traces, metrics, and logs
Don't Use OpenTelemetry When:
Building simple monolithic applications
Performance overhead is critical (consider sampling)
Team lacks observability infrastructure (Jaeger, Zipkin)
When to Use Prometheus
Need time-series metrics for monitoring and alerting
Building operational dashboards (Grafana)
Measuring SLIs for SLO compliance
Tracking business metrics (requests/sec, conversion rates)
Kubernetes/containerized environments
Don't Use Prometheus When:
Need high-cardinality metrics (Prometheus has limits)
Require long-term metric storage (use Thanos/Cortex)
Need push-based metrics (Prometheus is pull-based)
When to Use slog
Go 1.21+ projects (standard library, zero dependencies)
Need structured logging with JSON output
Want high-performance logging with minimal allocations
Integrating with log aggregation systems (Loki, ELK)
Go < 1.21 (use zap or zerolog instead)
Need complex log routing or filtering (use zap)
Require very specific features (audit trails, etc.)
Sampling Strategy Decision
Development/staging environments
Total traffic < 100 requests/sec
Debugging specific issues
Probabilistic Sampling When:
Production with moderate traffic (100-10K req/sec)
Sample rate: 1-10% typically
Tail-Based Sampling When:
High traffic production (>10K req/sec)
Only sample errors and slow requests
Requires tail-sampling processor (OpenTelemetry Collector)
Anti-Patterns to Avoid
❌ Not Propagating Context WRONG: Breaking trace context
func processOrder (order Order) error {
ctx := context.Background()
return validateOrder(ctx, order)
}
CORRECT: Propagate context through call chain
func processOrder (ctx context.Context, order Order) error {
return validateOrder(ctx, order)
}
❌ Cardinality Explosion WRONG: Unbounded label values
httpRequests.WithLabelValues(r.Method, r.URL.Path, userID).Inc()
CORRECT: Use bounded labels
httpRequests.WithLabelValues(r.Method, r.URL.Path).Inc()
❌ Logging Sensitive Data WRONG: Exposing PII and secrets
logger.Info("user login" ,
"email" , user.Email,
"password" , user.Password,
"token" , authToken,
)
CORRECT: Redact sensitive information
logger.Info("user login" ,
"user_id" , user.ID,
"method" , "password" ,
)
❌ Not Closing Spans func processOrder (ctx context.Context) error {
ctx, span := tracer.Start(ctx, "processOrder" )
if err := validate(); err != nil {
return err
}
return nil
}
CORRECT: Always defer span.End()
func processOrder (ctx context.Context) error {
ctx, span := tracer.Start(ctx, "processOrder" )
defer span.End()
if err := validate(); err != nil {
span.RecordError(err)
return err
}
return nil
}
❌ Synchronous Metric Export WRONG: Blocking requests with metric export
exporter := jaeger.New(jaeger.WithCollectorEndpoint(...))
tp := sdktrace.NewTracerProvider(
sdktrace.WithSyncer(exporter),
)
CORRECT: Use batching for async export
tp := sdktrace.NewTracerProvider(
sdktrace.WithBatcher(exporter),
)
❌ Missing Graceful Shutdown WRONG: Losing traces on shutdown
func main () "service" )
http.ListenAndServe(":8080" , nil )
}
CORRECT: Shutdown exporters properly
func main () "service" )
defer tp.Shutdown(context.Background())
server.ListenAndServe()
}
Best Practices
Context Propagation : Always pass context.Context through call chainsBounded Labels : Keep metric label cardinality under 1000 combinationsSampling : Use probabilistic sampling in high-traffic productionCorrelation IDs : Include trace_id in logs for correlationHealth Checks : Implement both /health (liveness) and /ready (readiness)Graceful Shutdown : Flush traces and metrics before exitError Recording : Use span.RecordError() for automatic error trackingMetric Naming : Follow Prometheus naming conventions (_total, _seconds)Log Levels : Use appropriate levels (Debug, Info, Warn, Error)Auto-Instrumentation : Use middleware for HTTP/gRPC when possible
Metric Naming Conventions Follow Prometheus best practices:
Counter Metrics (always increasing):
http_requests_total (not http_requests)payment_transactions_totalerrors_total Gauge Metrics (can go up or down):
active_connectionsqueue_sizememory_usage_bytes Histogram/Summary Metrics (observations):
http_request_duration_seconds (not _milliseconds)db_query_duration_secondsresponse_size_bytes
Use method, not http_method
Use status, not status_code or http_status
Use snake_case, not camelCase
Resources
golang-web-frameworks : HTTP server patterns and middlewaregolang-testing-strategies : Testing instrumented codeverification-before-completion : Validating observability setup
Quick Reference
Initialize OpenTelemetry tp, _ := initTracer("service-name" )
defer tp.Shutdown(context.Background())
Create Spans ctx, span := otel.Tracer("name" ).Start(ctx, "operation" )
defer span.End()
span.SetAttributes(attribute.String("key" , "value" ))
Define Metrics counter := promauto.NewCounterVec(opts, []string {"label" })
histogram := promauto.NewHistogramVec(opts, []string {"label" })
Structured Logging logger := slog.With("trace_id" , traceID)
logger.Info("message" , "key" , value)
Health Checks http.HandleFunc("/health" , livenessHandler)
http.HandleFunc("/ready" , readinessHandler)
Token Estimate : ~5,000 tokens (entry point + full content)
Version : 1.0.0
Last Updated : 2025-12-03
More from this repository
Related occupations SOC
Based on SOC occupation classification
