observability-patterns

name	observability-patterns
description	Observability patterns for Spring Boot — structured JSON logging (Logstash encoder), distributed tracing (Micrometer Tracing), custom Micrometer metrics, Prometheus alerting, and Grafana dashboards. Use when configuring logback-spring.xml, adding MDC correlation, creating @Timed/@Counted metrics, writing PromQL alert rules, setting up ELK/Loki log pipelines, or building Grafana dashboards for Spring Boot services.
triggers	{"natural":["structured logging","metrics","distributed tracing","health check","alerting"],"code":["Micrometer","@Timed","MeterRegistry","logback","MDC"]}
applicability	{"always":false,"triggers":{"files_match":["**/*Metrics*.java","**/*HealthCheck*.java","**/*Tracing*.java","**/logback*.xml","**/*Trace*.java"],"code_patterns":["MeterRegistry","@Observed","Tracer","MDC.put","LogstashEncoder","@Timed","@Counted"],"task_keywords":["metrics","logging","tracing","MDC","Micrometer","Prometheus","OpenTelemetry","health check","alerting"],"related_rules":["rules/java/observability.md"]}}
relevance_assessment	HIGH 80%+: new metric / log / trace / health probe OR alert rule MEDIUM 40-79%: production code that should emit telemetry but currently doesn't LOW 1-39%: existing instrumented code minor change ZERO: trivial fix, no production code, no SLO impact

Observability Patterns for Spring Boot

Structured Logging

Logback JSON (Production)

<springProfile name="!local">
  <appender name="JSON_CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
    <encoder class="net.logstash.logback.encoder.LogstashEncoder">
      <customFields>{"service":"order-service","env":"${SPRING_PROFILES_ACTIVE}"}</customFields>
    </encoder>
  </appender>
</springProfile>

Log Levels

Level	Use For
ERROR	Failures requiring attention (exceptions, data loss)
WARN	Recoverable issues (fallback triggered, retry)
INFO	Business events (order created, payment processed)
DEBUG	Technical detail (query params, cache hits) — local only

Rules

• SLF4J placeholders: log.info("Order created orderId={}", order.id()) — never concatenation.
• MDC for context: MDC.put("orderId", id). Clear in finally.
• Never log PII. Log identifiers only.

Reactive Context Propagation

public Mono<Order> createReactive(CreateOrderCommand cmd) {
    return Mono.deferContextual(ctx -> {
        MDC.put("traceId", ctx.getOrDefault("traceId", "none"));
        MDC.put("orderId", cmd.orderId());
        return orderRepository.save(Order.from(cmd))
            .doOnNext(o -> log.info("Order created orderId={}", o.id()))
            .doOnError(e -> log.error("Order creation failed", e))
            .doFinally(sig -> MDC.clear());
    });
}

Distributed Tracing

management:
  tracing:
    sampling:
      probability: 1.0  # 100% dev; 0.1 (10%) prod
  otlp:
    tracing:
      endpoint: ${OTEL_EXPORTER_OTLP_ENDPOINT:http://localhost:4318}/v1/traces

Custom spans: tracer.nextSpan().name("order.enrich").tag(...). Use WebClient.builder().observationRegistry(registry) for automatic trace propagation.

Custom Metrics

• Counter: Counter.builder("orders.created.total").register(meterRegistry) — totals
• Timer: Timer.builder("orders.creation.duration").register(meterRegistry).record(() -> work()) — latency
• Gauge: Gauge.builder("orders.pending.count", this, s -> s.count()).register(meterRegistry) — current value
• @Timed on controllers, @Observed on business methods

Prometheus Config

management:
  endpoints.web.exposure.include: health,metrics,prometheus
  endpoint.health.probes.enabled: true
  metrics.distribution:
    percentiles-histogram:
      http.server.requests: true
    slo:
      http.server.requests: 50ms,200ms,500ms,1s

Health Indicators & Kubernetes Probes

@Component
public class ExternalServiceHealthIndicator implements HealthIndicator {
    public Health health() {
        boolean ok = client.ping();
        return ok ? Health.up().build() : Health.down().withDetail("reason", "ping failed").build();
    }
}

# Kubernetes probes
livenessProbe:
  httpGet: { path: /actuator/health/liveness, port: 8080 }
  initialDelaySeconds: 30
  failureThreshold: 3
readinessProbe:
  httpGet: { path: /actuator/health/readiness, port: 8080 }
  initialDelaySeconds: 10
  failureThreshold: 3

Liveness = app alive (restart if fails). Readiness = can accept traffic (db, redis).

Alerting Thresholds

Metric	Condition	Severity
P99 latency > 500ms	5 min sustained	WARNING
P99 latency > 2s	2 min sustained	CRITICAL
Error rate > 1%	5 min sustained	WARNING
Error rate > 5%	1 min sustained	CRITICAL
JVM heap > 85%	10 min sustained	WARNING
DB pool saturation > 90%	Instant	CRITICAL
Health endpoint DOWN	Instant	CRITICAL

Rules

• Never log PII — mask emails, never log passwords/tokens/credentials
• SLF4J placeholders (log.info("x={}", x)) — never string concatenation
• Clear MDC in reactive doFinally blocks — prevent context leakage
• Set timeouts on health check external calls (2-5s) — prevent probe hangs

References

• references/logging.md — Full Logback config (local + JSON), reactive MDC propagation, WebFilter correlation, sensitive data rules
• references/tracing-metrics.md — Dependencies, custom spans, Observation API, Micrometer auto-instrumentation, Prometheus alerts, reactive health indicators
• references/detailed-patterns.md — Legacy combined reference (all patterns in one file)

Related Skills

• spring-webflux-patterns — WebFilter setup, production actuator defaults
• pentest — PII logging detection (OWASP A09)
• testing-workflow — Verification pipeline includes observability checks