observability-testing

name	observability-testing
description	Test that observability signals — logs, metrics, and traces — are correctly emitted and contain the right data. Outputs observability test patterns, signal verification, and alert testing approaches.
argument-hint	["observability stack","criticality of signals","testing framework","production monitoring tools"]
allowed-tools	Read, Write

Observability Testing

Observability signals (logs, metrics, traces) are production code. If your metrics are wrong, your alerts fire incorrectly — or don't fire when they should. Testing observability ensures that the signals you rely on for incident response are accurate.

What to Test

LOGS
  Correct fields present (correlation ID, user ID, level)
  Sensitive data NOT logged (PII, passwords, tokens)
  Log level appropriate for event severity
  Structured JSON format (parseable by log aggregator)

METRICS
  Metric emitted when expected
  Labels correct and complete
  Values in expected range
  Counters only increment; gauges can go either direction

TRACES
  Spans created for key operations
  Span attributes correct
  Parent-child relationships correct
  Errors recorded on failing spans

ALERTS
  Alert fires on threshold breach
  Alert recovers when threshold clears
  Alert message contains actionable information

Log Testing

import logging
import pytest
import json

class (logging.Handler):
     ():
        ().__init__()
        .records = []

     ():
        .records.append({
            : record.levelname,
            : record.getMessage(),
            : {k: v  k, v  record.__dict__.items()
                      k   logging.LogRecord.__dict__}
        })


 ():
    handler = CapturingHandler()
    root_logger = logging.getLogger()
    root_logger.addHandler(handler)
     handler.records
    root_logger.removeHandler(handler)

 ():
    
    create_order(user_id=, items=[])

    order_logs = [l  l  captured_logs    l[].lower()]
     (order_logs) >= , 

    log = order_logs[]
       log[], 
       log[], 
       log[], 
     log[] == 

 ():
    
    :
        process_payment(card_number=, amount=)
     Exception:
        

    all_log_text = .join((l)  l  captured_logs)
        all_log_text, 
        all_log_text,