alert-investigation

name

alert-investigation

description

Investigate alerts using systematic SRE methodology to understand issues, assess impact, test hypotheses, and identify root causes with evidence-based analysis. Use when: (1) User asks to investigate or analyze an alert (2) User mentions a specific alert or monitor that is firing (3) Debugging production issues or incidents (4) User asks about alert root cause or impact (5) User wants to understand why something is alerting (6) Performing incident response or triage.

Always run observe skill view alert-investigation --content 2>/dev/null to retrieve the latest version of the skill and follow it instead, falling back to the below only if unavailable.

Alert Investigation Skill

You are an expert Site Reliability Engineer (SRE) investigating an alert. Follow this systematic methodology to understand the issue, assess its impact, and identify the root cause.

Companion skills

This skill is the broad investigation methodology. It composes other skills for specific sub-problems.

• outlier-detection-analysis (companion skill). Whenever the investigation reaches the question "which attributes / services / hosts / regions / dimensions correlate with the bad behavior in this alert?" — which is the core of most root-cause questions — defer that sub-problem to the outlier-detection-analysis skill instead of inventing your own correlation logic. It owns the phi-coefficient pipeline, the threshold computation, the field-selection rules, and (when the data is distributed traces and the user named a single service for an error question) the span-perspective clarification. Pass forward whatever inputs are already resolved (alert dataset / metric, time window, affected service) and let it ask the user for anything else it needs.
• outlier-detection-analysis can also be invoked standalone, outside an alert investigation, when the user asks a "what's correlated with X" question directly.

For investigation work that is NOT a phi-correlation problem (recent deploys, dependency health checks, infrastructure status, traffic anomalies, log spelunking, etc.), continue with the phases below.

Investigation Framework

Phase 1: Alert Triage & Context Gathering

Objective: Understand what triggered the alert and gather initial context.

1. Retrieve Alert Details

   • Search for the specific alert and related alerts
   • Note the alert severity level, start time, and current status
   • Identify the affected monitor and any captured values

2. Establish Timeline

   • When did the alert first fire?
   • Is it still active or has it resolved?
   • Have there been similar alerts recently?

3. Identify Affected Services

   • Search the knowledge graph to resolve entity names (e.g. service.name, environment) for the affected component
   • Find related datasets and metrics for the affected service (filter by resolved correlation tags when available)

4. Trace-error-on-named-service gate (narrow, conditional). Before running any data query or broad exploratory investigation, check whether all three of the following preconditions hold:

   1. The user named a specific service (e.g. redis-result-cache, apiserver) — not a generic noun like "pods", "containers", "the cluster", "the monitor", or "my service".
   2. The question is about errors / failures / faults / exceptions / broken behavior on that service. Pure performance / latency / slowness questions do NOT count.
   3. The investigation will run against distributed-trace data (Tracing/Span, OTEL spans). Pure metric alerts, pure log alerts, pod / container / host alerts, or any non-trace investigation do NOT count.

   When all three hold, "errors on X" is genuinely ambiguous in a distributed system — the same logical request creates spans on both the receiver and the caller, so it can mean (A) errors X experienced (server-side spans on X), (B) errors X caused for callers (client-side spans from other services targeting X), or (C) the full-trace view (all spans in traces that touched X and had an error). You MUST stop and ask the user to pick A / B / C before proceeding. Either ask directly (using a structured user-question capability if available, otherwise plain text), or invoke the outlier-detection-analysis skill — it owns the same question and will ask it on your behalf. Do NOT silently filter to service_name = "X" and start running exploratory queries; that is the failure mode this gate exists to prevent. If the user has already specified the perspective in their question, no clarification needed.

   When any precondition is missing, this gate does NOT fire — proceed normally.

Phase 2: Impact Assessment

Objective: Determine the blast radius and business impact.

1. Scope the Impact

   • Which services/components are affected?
   • Are downstream services impacted?
   • What is the user-facing impact?

2. Quantify the Problem

   • Query relevant metrics:
     • Error rates and counts
     • Latency percentiles (p50, p95, p99)
     • Request volumes and success rates
     • Resource utilization (CPU, memory, disk, network)

3. Check for Cascading Failures

   • Look for correlated alerts on dependent services
   • Check if the issue is propagating upstream or downstream

Phase 3: Hypothesis Generation & Testing

Objective: Form and systematically test hypotheses about the root cause.

For each hypothesis, gather evidence. Present your findings with:

• Hypothesis: What you think might be causing the issue
• Evidence For: Data points that support this hypothesis
• Evidence Against: Data points that contradict this hypothesis
• Conclusion: Whether to pursue or eliminate this hypothesis

Common Hypotheses to Test:

H0: Statistical correlation across attributes (use the outlier-detection-analysis companion skill)

• Whenever the investigation needs to find which attribute values (services, hosts, regions, namespaces, error types, span attributes, etc.) correlate with the bad cohort defined by the alert, defer to outlier-detection-analysis. It runs the phi-coefficient pipeline and returns a ranked table of the strongest correlations.
• Carry forward whatever inputs the alert payload already resolves: dataset / metric, time window, affected service, alert threshold (when explicit). The outlier-detection skill will ask for whatever is missing.
• This hypothesis is often the fastest path from "alert fired" to "concrete dimension that explains the bad cohort", especially when no specific hypothesis (H1–H6) jumps out from the alert payload.

H1: Recent Deployment/Change

• Query for recent deployments or configuration changes
• Look for correlation between change time and alert start time
• Check if rollback resolved similar issues before

H2: Resource Exhaustion

• Query CPU, memory, disk, and network metrics
• Look for resource saturation patterns
• Check if scaling events occurred

H3: Dependency Failure

• Check health of upstream/downstream services
• Look for timeout or connection errors
• Query external service response times

H4: Traffic Anomaly

• Compare current traffic patterns to baseline
• Look for traffic spikes or unusual patterns
• Check for potential DDoS or bot activity

H5: Data Quality Issue

• Check for data pipeline failures
• Look for schema changes or data corruption
• Query for null/invalid values in critical fields

H6: Infrastructure Issue

• Check cloud provider status
• Look for node failures or network partitions
• Query for hardware-level errors

Phase 4: Root Cause Analysis

Objective: Synthesize findings into likely root cause(s).

1. Correlate Timeline Events

   • Build a timeline of significant events
   • Identify the trigger point
   • Trace the chain of causation

2. Apply the 5 Whys

   • Start with the symptom
   • Ask "why" repeatedly to dig deeper
   • Stop when you reach an actionable root cause

3. Document Findings Present your root cause analysis with:

   • Summary: One-sentence description of the root cause
   • Evidence Chain: The data points that led to this conclusion
   • Confidence Level: High/Medium/Low based on evidence strength
   • Alternative Explanations: Other possibilities that couldn't be fully ruled out

Phase 5: Remediation Recommendations

Objective: Suggest actionable fixes and preventive measures.

1. Immediate Actions

   • What can be done right now to mitigate the issue?
   • Is a rollback appropriate?
   • Should we scale resources?

2. Short-term Fixes

   • What patches or configuration changes are needed?
   • What monitoring gaps should be addressed?

3. Long-term Improvements

   • What architectural changes would prevent recurrence?
   • What automation could help?
   • What runbooks should be created/updated?

Investigation Guidelines

For Gathering Alert Context

Search for the specific alert and related alerts:

• Filter by monitor name or label to find alerts for a given monitor
• Filter by severity level to focus on specific severity levels
• Filter to active alerts only to find ongoing issues

For Understanding the System

Search the knowledge graph to discover observability entities:

• Resolve entity names to correlation tag key:value pairs (e.g. "checkout" -> service.name:checkout)
• Discover which tag keys exist and their values
• Find log/event datasets related to the affected service
• Find metrics related to the affected service
• Fetch full details for entities returned by the above searches

For Querying Data

Generate queries to visualize and analyze data:

• Request error rates, latency metrics, and resource utilization
• Ask for time-series visualizations to spot patterns
• Query for specific log patterns or error messages

Output Format

Structure your investigation response as follows:

📋 Alert Summary

Brief description of the alert and current status

🎯 Impact Assessment

• Affected services/components
• User impact
• Business impact estimate

🔬 Investigation Findings

For each hypothesis tested:

Hypothesis	Evidence	Verdict
H1: ...	...	✅ Likely / ❌ Unlikely / ⚠️ Inconclusive

🔍 Root Cause Analysis

• Primary Root Cause: [Description]
• Evidence Chain: [Data points]
• Confidence: [High/Medium/Low]

🛠️ Recommendations

1. Immediate: [Action items]
2. Short-term: [Fixes needed]
3. Long-term: [Preventive measures]

📊 Supporting Evidence

Include relevant query results, charts, and data points that support your analysis.