메뉴
Comprehensive observability strategy including metrics, logs, traces, and alerting. Use when setting up new applications, debugging production issues, performance optimization, SLA/SLO definition, incident response, or establishing monitoring infrastructure.
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
SOC 직업 분류 기준
Project-agnostic workflow guidance for Backlog.md MCP task management. Covers search-first protocol, scope assessment, task creation standards, lifecycle management, and cross-session coordination.
Backlog management with the bd (beads) utility for issue tracking. Use when creating issues, claiming work, updating task status, closing completed work, or syncing the backlog with git.
How to create new Agent Skills following the open standard. Use when authoring new skills, extending agent capabilities, or packaging specialized knowledge into reusable formats.
Project-agnostic skill for git commit hygiene, linking tasks to commits, and maintaining clean repository state.
Git hygiene for multi-agent collaborative work. Use when performing version control operations, syncing with remote, committing changes, or ensuring work is properly shared with the team.
Project-agnostic guidance for continuing work across Claude sessions. Ensures context recovery, task resumption, and state reconciliation.
| name | monitoring |
| description | Comprehensive observability strategy including metrics, logs, traces, and alerting. Use when setting up new applications, debugging production issues, performance optimization, SLA/SLO definition, incident response, or establishing monitoring infrastructure. |
Measure what matters, alert on what's actionable.
You can't improve what you don't measure. Observability is not about collecting all possible data—it's about collecting the right data to understand system health, debug issues quickly, and make informed decisions. Alert on symptoms that require action, not on every fluctuation.
Use this skill when:
Modern observability is built on three complementary pillars:
┌─────────────────────────────────────────────┐
│ OBSERVABILITY │
├─────────────┬──────────────┬────────────────┤
│ METRICS │ LOGS │ TRACES │
├─────────────┼──────────────┼────────────────┤
│ What is │ What │ Where is │
│ happening? │ happened? │ the problem? │
│ │ │ │
│ Time-series │ Event │ Request flow │
│ data │ records │ through system │
│ │ │ │
│ Aggregated │ Detailed │ Distributed │
│ numbers │ context │ context │
└─────────────┴──────────────┴────────────────┘
Time-series numerical data aggregated over time.
Examples:
Characteristics:
When to Use:
Timestamped event records with contextual information.
Examples:
{
"timestamp": "2025-01-10T14:32:15Z",
"level": "ERROR",
"service": "payment-service",
"trace_id": "abc123",
"user_id": "user_456",
"message": "Payment processing failed",
"error": "Gateway timeout",
"amount": 99.99,
"currency": "USD"
}
Characteristics:
When to Use:
Request flow tracking across distributed systems.
User Request → API Gateway → Auth Service → User Service → Database
↓
Payment Service → Payment Gateway
↓
Email Service → Email Provider
Example Trace:
Trace ID: abc123
Total Duration: 450ms
Span 1: API Gateway [0-450ms] ████████████████████
Span 2: Auth Service [10-30ms] ██
Span 3: User Service [35-100ms] ██████
Span 4: Database Query [40-95ms] █████
Span 5: Payment Service [105-400ms] ████████████████ ← SLOW!
Span 6: Payment Gateway [120-390ms] ███████████████
Span 7: Email Service [405-440ms] ███
Characteristics:
When to Use:
Google's SRE framework for monitoring any system.
How long does it take to service a request?
Key Metrics:
Why Percentiles Matter:
Average: 100ms ← Hides outliers!
p50: 80ms ← Half of requests
p95: 200ms ← 95% of requests
p99: 500ms ← 99% of requests (worst experience)
p999: 2000ms ← Worst 0.1% (but affects real users!)
Alert Example:
IF p99_response_time > 1000ms for 5 minutes
THEN alert: "High latency affecting 1% of users"
How much demand is being placed on the system?
Key Metrics:
Why It Matters:
Alert Example:
IF requests_per_second > 10000 for 10 minutes
THEN alert: "Unusual traffic spike detected"
What is the rate of failing requests?
Key Metrics:
Error Budget:
SLO: 99.9% availability
Error budget per month: 0.1% = 43 minutes downtime
If error rate = 1% for 1 hour
→ Consumed 36 minutes of error budget
→ Only 7 minutes remaining this month!
Alert Example:
IF error_rate > 1% for 5 minutes
THEN page: "Critical: High error rate"
How full is the service?
Key Metrics:
Why It Matters:
Alert Example:
IF memory_usage > 90% for 10 minutes
THEN alert: "Memory saturation warning"
Best for request-driven services (APIs, web services).
Example Dashboard:
Service: payment-api
Rate: 1,250 req/sec ▲ 15%
Errors: 12.5 req/sec ▲ 200% ⚠️
Duration:
p50: 45ms
p95: 120ms
p99: 280ms
Best for infrastructure and hardware monitoring.
Example:
CPU:
Utilization: 75%
Saturation: Load average 4.2 (4 cores)
Errors: 0
Disk:
Utilization: 60%
Saturation: I/O queue depth: 12
Errors: 3 read errors
Pull-based metrics collection and time-series database.
# prometheus.yml
scrape_configs:
- job_name: 'myapp'
scrape_interval: 15s
static_configs:
- targets: ['localhost:8080']
Application Instrumentation (Python):
from prometheus_client import Counter, Histogram, start_http_server
import time
# Define metrics
request_count = Counter('http_requests_total', 'Total HTTP requests', ['method', 'endpoint'])
request_duration = Histogram('http_request_duration_seconds', 'HTTP request duration')
# Instrument code
@request_duration.time()
def handle_request(method, endpoint):
request_count.labels(method=method, endpoint=endpoint).inc()
# Handle request...
# Expose metrics endpoint
start_http_server(8000) # Metrics at http://localhost:8000/metrics
PromQL Queries:
# Request rate (requests per second)
rate(http_requests_total[5m])
# Error rate (percentage)
rate(http_requests_total{status=~"5.."}[5m])
/ rate(http_requests_total[5m]) * 100
# p95 latency
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))
# CPU usage by pod
container_cpu_usage_seconds_total{pod=~"myapp-.*"}
Benefits:
Drawbacks:
Visualization and dashboarding for metrics.
Features:
Example Dashboard JSON:
{
"title": "Service Overview",
"panels": [
{
"title": "Request Rate",
"targets": [{
"expr": "rate(http_requests_total[5m])"
}],
"type": "graph"
},
{
"title": "Error Rate",
"targets": [{
"expr": "rate(http_requests_total{status=~\"5..\"}[5m]) / rate(http_requests_total[5m]) * 100"
}],
"alert": {
"conditions": [{
"evaluator": {"type": "gt", "params": [1]},
"query": {"params": ["A", "5m", "now"]}
}]
}
}
]
}
AWS-native metrics, logs, and alarms.
# Python: Publishing custom metrics
import boto3
cloudwatch = boto3.client('cloudwatch')
cloudwatch.put_metric_data(
Namespace='MyApp',
MetricData=[
{
'MetricName': 'RequestCount',
'Value': 1,
'Unit': 'Count',
'Timestamp': datetime.utcnow(),
'Dimensions': [
{'Name': 'Environment', 'Value': 'production'},
{'Name': 'Service', 'Value': 'api'}
]
}
]
)
Benefits:
Drawbacks:
All-in-one observability platform.
Features:
# Python: Datadog StatsD
from datadog import initialize, statsd
initialize(api_key='YOUR_API_KEY')
# Increment counter
statsd.increment('myapp.requests', tags=['endpoint:/api/users'])
# Record timing
statsd.timing('myapp.request_duration', 250)
# Set gauge
statsd.gauge('myapp.queue_depth', 42)
Benefits:
Drawbacks:
Open-source log aggregation and analysis.
Application → Logstash → Elasticsearch → Kibana
(collect) (store/index) (visualize)
Structured Logging Example:
import logging
import json
class JsonFormatter(logging.Formatter):
def format(self, record):
log_data = {
'timestamp': self.formatTime(record),
'level': record.levelname,
'message': record.getMessage(),
'service': 'myapp',
'trace_id': getattr(record, 'trace_id', None)
}
return json.dumps(log_data)
handler = logging.StreamHandler()
handler.setFormatter(JsonFormatter())
logging.root.addHandler(handler)
Logstash Configuration:
input {
file {
path => "/var/log/myapp/*.log"
codec => "json"
}
}
filter {
if [level] == "ERROR" {
mutate {
add_tag => ["error"]
}
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "myapp-logs-%{+YYYY.MM.dd}"
}
}
Log aggregation designed like Prometheus (labels, not full-text indexing).
Benefits:
Promtail Configuration:
server:
http_listen_port: 9080
clients:
- url: http://loki:3100/loki/api/v1/push
scrape_configs:
- job_name: myapp
static_configs:
- targets:
- localhost
labels:
job: myapp
__path__: /var/log/myapp/*.log
Querying Logs in Grafana:
{job="myapp"} |= "error" | json | level="ERROR"
Enterprise log management platform.
Features:
Drawbacks:
Distributed tracing system developed by Uber.
OpenTelemetry Instrumentation (Python):
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.jaeger.thrift import JaegerExporter
# Setup
trace.set_tracer_provider(TracerProvider())
jaeger_exporter = JaegerExporter(
agent_host_name="localhost",
agent_port=6831,
)
trace.get_tracer_provider().add_span_processor(
BatchSpanProcessor(jaeger_exporter)
)
tracer = trace.get_tracer(__name__)
# Usage
def process_order(order_id):
with tracer.start_as_current_span("process_order") as span:
span.set_attribute("order_id", order_id)
with tracer.start_as_current_span("validate_order"):
validate(order_id)
with tracer.start_as_current_span("charge_payment"):
charge(order_id)
with tracer.start_as_current_span("send_confirmation"):
send_email(order_id)
Vendor-neutral instrumentation standard.
Benefits:
Auto-Instrumentation (Python):
# Install
pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install
# Run with auto-instrumentation
opentelemetry-instrument \
--traces_exporter otlp \
--metrics_exporter otlp \
--service_name myapp \
python app.py
AWS-native distributed tracing.
from aws_xray_sdk.core import xray_recorder
from aws_xray_sdk.core import patch_all
# Patch libraries
patch_all()
# Manual instrumentation
@xray_recorder.capture('process_order')
def process_order(order_id):
xray_recorder.current_segment().put_annotation('order_id', order_id)
# Process order...
Bad Alerting:
3:00 AM: Disk usage 71% ⚠️
3:15 AM: Memory usage 82% ⚠️
3:30 AM: CPU spike to 90% for 10 seconds ⚠️
3:45 AM: Database connection pool 70% full ⚠️
4:00 AM: Disk usage 72% ⚠️
Result: On-call engineer ignores/silences all alerts. Real issues get missed.
Good Alerting:
3:00 AM: [CRITICAL] Error rate 15% for 10 minutes - users affected!
Result: On-call engineer immediately investigates actionable issue.
Use 3-4 severity levels, no more.
❌ Bad: Cause-based alert
IF cpu_usage > 80% THEN alert
Problem: High CPU might not affect users. Not actionable.
✅ Good: Symptom-based alert
IF p99_latency > 1000ms AND error_rate > 1% THEN alert
Why: Users are actually affected. Clear action needed.
Every alert must answer: "What should I do about this?"
❌ Bad Alert:
ALERT: Database queries slow
✅ Good Alert:
ALERT: Database p99 query time > 5s for 10 minutes
User Impact: Checkout page timing out for 1% of users
Runbook: https://wiki.company.com/runbooks/slow-database
Dashboard: https://grafana.company.com/d/database
Recent Changes: Last deploy 2 hours ago (v1.2.3)
Immediate Actions:
1. Check slow query log
2. Check for locks: SELECT * FROM pg_locks WHERE granted=false
3. Consider rolling back deploy if started after v1.2.3 deploy
Error Budget Concept:
SLO: 99.9% availability = 99.9% successful requests
Error Budget: 0.1% of requests can fail
Monthly Error Budget:
- 0.1% of 100M requests = 100,000 failed requests allowed
- 43 minutes of downtime allowed
Alert when error budget consumption rate too high:
IF current_error_rate will exhaust error_budget in < 7 days
THEN alert: "Error budget burn rate too high"
Use alert grouping and deduplication:
# Alertmanager (Prometheus)
route:
group_by: ['alertname', 'cluster', 'service']
group_wait: 10s
group_interval: 10s
repeat_interval: 12h
inhibit_rules:
# Don't alert on high latency if service is down
- source_match:
severity: 'critical'
alertname: 'ServiceDown'
target_match:
severity: 'warning'
alertname: 'HighLatency'
equal: ['service']
Deploy Alert → Monitor False Positive Rate
↓
Too many false positives?
↙ ↘
YES NO
↓ ↓
Adjust thresholds Keep alert
Adjust duration
Add conditions
↓
Re-deploy alert
↓
Monitor
Target:
A metric that measures service quality.
Examples:
Good SLI:
SLI = (successful requests) / (total requests)
= 99,500 / 100,000
= 99.5%
Target value or range for an SLI.
Examples:
SLO Structure:
SLO: [percentage] of [what] in [time period]
Example:
SLO: 99.9% of API requests succeed in a rolling 30-day window
Contract with users about service levels, often with consequences.
Example:
SLA: 99.9% uptime monthly
Uptime: 99.9% - 100% → No credit
Uptime: 99.0% - 99.9% → 10% credit
Uptime: 95.0% - 99.0% → 25% credit
Uptime: < 95.0% → 50% credit
SLA ≥ SLO > SLI (actual)
Example:
SLA: 99.5% (contractual minimum)
SLO: 99.9% (internal target - buffer above SLA)
SLI: 99.95% (actual measurement)
Buffer = SLA - SLO = 0.4% = error budget
100% is:
Instead:
Step 1: Measure current SLI for 30 days
Current: 99.7% success rate
Step 2: Set SLO slightly below current (buffer)
SLO: 99.5%
Step 3: Monitor and adjust quarterly
If easily meeting SLO → Consider raising
If struggling → Investigate systemic issues
Purpose: Quickly answer "Is everything OK?"
Panels:
Layout:
┌─────────────────────────────────────────────┐
│ System Health: GREEN ✓ │
├──────────────┬──────────────┬───────────────┤
│ Latency │ Traffic │ Errors │
│ p99: 250ms │ 1.2K req/s │ 0.05% │
│ ████░░ │ ██████░░ │ ░░░░░░ │
├──────────────┴──────────────┴───────────────┤
│ Recent Deployments │
│ ──●────●──────────●─── (last 24h) │
│ │
│ Active Alerts: 0 │
└──────────────────────────────────────────────┘
Purpose: Understand specific service performance.
Panels:
Purpose: Connect technical metrics to business outcomes.
Examples:
Why It Matters: Technical outages should correlate with business impact:
Error rate spike at 2:00 PM
↓
Transactions dropped 80%
↓
Revenue loss: $10,000/hour
↓
Priority: CRITICAL
Purpose: Information needed during active incidents.
Panels:
Add monitoring from day one, not after issues arise.
# Bad: No instrumentation
def process_payment(amount):
return gateway.charge(amount)
# Good: Instrumented
@metrics.timer('payment_duration')
def process_payment(amount):
with tracer.start_span('process_payment'):
try:
result = gateway.charge(amount)
metrics.increment('payment_success')
return result
except Exception as e:
metrics.increment('payment_failure')
logger.error('Payment failed', extra={
'amount': amount,
'error': str(e),
'trace_id': get_trace_id()
})
raise
❌ Bad: Unstructured
logger.info(f"User {user_id} purchased {item} for ${amount}")
Problem: Hard to parse, query, and alert on.
✅ Good: Structured (JSON)
logger.info("purchase_completed", extra={
'event': 'purchase',
'user_id': user_id,
'item_id': item,
'amount': amount,
'currency': 'USD',
'trace_id': trace_id
})
Benefit: Easy to filter, aggregate, and build alerts.
Connect metrics → logs → traces using trace ID.
import uuid
def handle_request():
trace_id = str(uuid.uuid4())
# Add to metrics
metrics.increment('requests', tags={'trace_id': trace_id})
# Add to logs
logger.info('Processing request', extra={'trace_id': trace_id})
# Add to response headers (for debugging)
return response, {'X-Trace-ID': trace_id}
Debugging Flow:
1. User reports slow request
2. Find trace_id in response headers
3. Search logs: trace_id="abc123"
4. View trace in Jaeger: abc123
5. Identify slow span: database query took 5s
6. Fix query performance
Your service is only as reliable as its dependencies.
# Monitor external API
@metrics.timer('external_api_duration')
def call_external_api():
try:
response = requests.get('https://api.external.com', timeout=5)
metrics.increment('external_api_success')
return response
except requests.Timeout:
metrics.increment('external_api_timeout')
raise
except requests.RequestException:
metrics.increment('external_api_error')
raise
Dashboard:
Dependencies Health:
Database: 99.9% success ✓
Redis Cache: 99.95% success ✓
Payment API: 97.2% success ⚠️ ← Issue!
Email API: 99.8% success ✓
Enable filtering and aggregation.
# Good: Tagged metrics
metrics.increment('http_requests', tags={
'method': 'POST',
'endpoint': '/api/users',
'status': '200',
'environment': 'production'
})
# Now you can query:
# - All POST requests
# - All /api/users requests
# - All 5xx errors
# - Production vs staging comparison
❌ Bad: Unique value per request
metrics.increment('requests', tags={
'user_id': user_id # Millions of unique values!
})
Problem: Creates millions of time series, explodes storage costs.
✅ Good: Low cardinality
metrics.increment('requests', tags={
'user_tier': 'premium' # Only 3-4 unique values
})
Developer writes code
↓
Add instrumentation (metrics, logs, traces)
↓
Define local monitoring (docker-compose)
↓
Test monitoring locally
↓
Commit code with instrumentation
Code deployed to staging
↓
Verify metrics appear in Prometheus/Datadog
↓
Create/update Grafana dashboards
↓
Define alerts in Alertmanager
↓
Test alerts (trigger conditions manually)
↓
Deploy to production
↓
Monitor dashboards during deployment
Monitor dashboards daily
↓
Alert fires → Investigate
↓
Use logs and traces to debug
↓
Fix issue
↓
Update runbooks
↓
Tune alert if false positive
Application:
Infrastructure:
Dependencies:
Business:
Service: [service-name]
SLO: [percentage]% of [what] in [time period]
Example:
Service: payment-api
SLO: 99.9% of API requests succeed in a rolling 30-day window
Measurement:
SLI: (successful_requests / total_requests) * 100
Window: 30 days rolling
Target: 99.9%
Error Budget:
Allowed failures: 0.1% = ~43 minutes downtime/month
Current status: 99.95% (well within budget)
Alerts:
- Error budget burn rate > 5x: CRITICAL (page on-call)
- Error budget < 20% remaining: WARNING (notify team)
# Prometheus
# Query current request rate
curl 'http://localhost:9090/api/v1/query?query=rate(http_requests_total[5m])'
# Grafana
# Create API key
curl -X POST http://admin:admin@localhost:3000/api/auth/keys \
-H "Content-Type: application/json" \
-d '{"name":"monitoring","role":"Viewer"}'
# Jaeger
# Query traces by service
curl 'http://localhost:16686/api/traces?service=myapp&limit=10'
# CloudWatch
# Get metric statistics
aws cloudwatch get-metric-statistics \
--namespace AWS/EC2 \
--metric-name CPUUtilization \
--start-time 2025-01-10T00:00:00Z \
--end-time 2025-01-10T23:59:59Z \
--period 3600 \
--statistics Average
Remember: Good monitoring is invisible when everything works, but invaluable when things break. Instrument early, alert sparingly, and always connect metrics to user impact. Measure what matters, not just what's easy to measure.