Implement comprehensive observability for service meshes including distributed tracing, metrics, and visualization. Use when setting up mesh monitoring, debugging latency issues, or implementing SL...
Implement comprehensive observability for service meshes including distributed tracing, metrics, and visualization. Use when setting up mesh monitoring, debugging latency issues, or implementing SL...
risk
unknown
source
community
date_added
2026-02-27
Service Mesh Observability
Complete guide to observability patterns for Istio, Linkerd, and service mesh deployments.
Do not use this skill when
The task is unrelated to service mesh observability
You need a different domain or tool outside this scope
Instructions
Clarify goals, constraints, and required inputs.
Apply relevant best practices and validate outcomes.
Provide actionable steps and verification.
If detailed examples are required, open resources/implementation-playbook.md.
# Request rate by service
sum(rate(istio_requests_total{reporter="destination"}[5m])) by (destination_service_name)
# Error rate (5xx)
sum(rate(istio_requests_total{reporter="destination", response_code=~"5.."}[5m]))
/ sum(rate(istio_requests_total{reporter="destination"}[5m])) * 100
# P99 latency
histogram_quantile(0.99,
sum(rate(istio_request_duration_milliseconds_bucket{reporter="destination"}[5m]))
by (le, destination_service_name))
# TCP connections
sum(istio_tcp_connections_opened_total{reporter="destination"}) by (destination_service_name)
# Request size
histogram_quantile(0.99,
sum(rate(istio_request_bytes_bucket{reporter="destination"}[5m]))
by (le, destination_service_name))
Template 3: Jaeger Distributed Tracing
# Jaeger installation for IstioapiVersion:install.istio.io/v1alpha1kind:IstioOperatorspec:meshConfig:enableTracing:truedefaultConfig:tracing:sampling:100.0# 100% in dev, lower in prodzipkin:address:jaeger-collector.istio-system:9411---# Jaeger deploymentapiVersion:apps/v1kind:Deploymentmetadata:name:jaegernamespace:istio-systemspec:selector:matchLabels:app:jaegertemplate:metadata:labels:app:jaegerspec:containers:-name:jaegerimage:jaegertracing/all-in-one:1.50ports:-containerPort:5775# UDP-containerPort:6831# Thrift-containerPort:6832# Thrift-containerPort:5778# Config-containerPort:16686# UI-containerPort:14268# HTTP-containerPort:14250# gRPC-containerPort:9411# Zipkinenv:-name:COLLECTOR_ZIPKIN_HOST_PORTvalue:":9411"
Template 4: Linkerd Viz Dashboard
# Install Linkerd viz extension
linkerd viz install | kubectl apply -f -
# Access dashboard
linkerd viz dashboard
# CLI commands for observability# Top requests
linkerd viz top deploy/my-app
# Per-route metrics
linkerd viz routes deploy/my-app --to deploy/backend
# Live traffic inspection
linkerd viz tap deploy/my-app --to deploy/backend
# Service edges (dependencies)
linkerd viz edges deployment -n my-namespace
Template 5: Grafana Dashboard JSON
{"dashboard":{"title":"Service Mesh Overview","panels":[{"title":"Request Rate","type":"graph","targets":[{"expr":"sum(rate(istio_requests_total{reporter=\"destination\"}[5m])) by (destination_service_name)","legendFormat":"{{destination_service_name}}"}]},{"title":"Error Rate","type":"gauge","targets":[{"expr":"sum(rate(istio_requests_total{response_code=~\"5..\"}[5m])) / sum(rate(istio_requests_total[5m])) * 100"}],"fieldConfig":{"defaults":{"thresholds":{"steps":[{"value":0,"color":"green"},{"value":1,"color":"yellow"},{"value":5,"color":"red"}]}}}},{"title":"P99 Latency","type":"graph","targets":[{"expr":"histogram_quantile(0.99, sum(rate(istio_request_duration_milliseconds_bucket{reporter=\"destination\"}[5m])) by (le, destination_service_name))","legendFormat":"{{destination_service_name}}"}]},{"title":"Service Topology","type":"nodeGraph","targets":[{"expr":"sum(rate(istio_requests_total{reporter=\"destination\"}[5m])) by (source_workload, destination_service_name)"}]}]}}
Template 6: Kiali Service Mesh Visualization
# Kiali installationapiVersion:kiali.io/v1alpha1kind:Kialimetadata:name:kialinamespace:istio-systemspec:auth:strategy:anonymous# or openid, tokendeployment:accessible_namespaces:-"**"external_services:prometheus:url:http://prometheus.istio-system:9090tracing:url:http://jaeger-query.istio-system:16686grafana:url:http://grafana.istio-system:3000