incidents

name

incidents

description

Triage and manage cortex-detected incidents — unacknowledged error signatures, recent notification firings, similar past incidents, and full incident context around a time window. Use whenever the user asks "what errors haven't been addressed", "what's still firing", "have we seen this before", "ack this error", "acknowledge this signature", "what notifications went out recently", or wants a full picture of what happened around an incident.

Cortex Incident Management

Triage the error-signature backlog, review recent alert firings, find prior occurrences of a problem, and pull full context around a time window. This skill is for incident-response workflows — not general log search (use cortex for that) and not connection/service failures (use troubleshoot for that).

Workflow

1. Find what's unaddressed

Start here for "what's still open" / "what haven't we fixed" questions.

cortex action=unaddressed_errors [limit=N] [include_acknowledged=true] (CLI: cortex alerts signatures [--include-acknowledged] [--limit N])

Returns signatures[], each with a signature_hash (the stable key for everything downstream), a template/sample_message, severity, sample_hostname, first_seen_at/last_seen_at, total_count, and count_last_1h. By default only unacknowledged signatures are returned — pass include_acknowledged=true to see the full set including ones already handled.

Triage priority: high count_last_1h (still actively firing) beats high total_count alone (could be a stale, no-longer-occurring pattern).

2. Acknowledge or un-acknowledge a signature

Once a signature is understood and either fixed or explicitly accepted as known/benign, acknowledge it so it drops out of the default unaddressed_errors view:

cortex action=ack_error signature_hash=<hash> [notes="..."] (CLI: cortex alerts signatures ack <signature_hash> [--notes TEXT])

To reverse a bad acknowledgement (e.g. it turned out not to be fixed):

cortex action=unack_error signature_hash=<hash> [reason="..."] (CLI: cortex alerts signatures unack <signature_hash> [--reason TEXT])

Both are admin-scoped actions — they require cortex:admin (static bearer tokens need CORTEX_STATIC_TOKEN_ADMIN=true). Always state which signature you're acknowledging and why before calling ack_error — this is a state-changing action, not a read.

3. Check what's already been notified

cortex action=notifications_recent [rule_id=... ] [since=...] [limit=N] (CLI: cortex alerts notifications [--rule-id ID] [--since TIME] [--limit N])

Use this to answer "did we already get paged for this" before re-raising something, or to confirm a notification rule actually fired when expected.

4. Check whether this has happened before

cortex action=similar_incidents query="<free text>" [host=...] [app=...] [severity_min=...] [since=...] [until=...] [window_minutes=N] [limit=N] (CLI: cortex sessions similar "<query>" [--host H] [--app A] [--severity-min S] [--since T] [--until T] [--window-minutes N] [--limit N])

This clusters matching logs into IncidentClusters (window_start/ window_end, log_count, severity_peak, representative_messages, correlated AI sessions if any). Use the error's template/sample_message from step 1 as the query when following up on a specific signature.

5. Pull full context around a window

cortex action=incident_context since=<ISO> until=<ISO> [host=...] [app=...] [query=...] [severity_min=...] [limit=N] (CLI: cortex sessions incidentcontext --since <ISO> --until <ISO> [--host H] [--app A] [--query Q] [--severity-min S] [--limit N])

Both since and until are required — there is no incident-id concept that threads automatically from the earlier steps into this one. Build the window yourself: pad a signature's first_seen_at/last_seen_at (step 1) or a cluster's window_start/window_end (step 4) by a few minutes on each side, then pass that as since/until here.

Returns severity/app breakdowns for the window plus the actual error_logs and any correlated ai_sessions — this is the "show me everything that was happening" view once you already know roughly when and where.

Composing the workflow

A typical incident triage session chains these in sequence: unaddressed_errors to find a candidate signature → similar_incidents with that signature's message to see if it's recurring → incident_context with a padded window from either result to see the full blast radius → ack_error once you've confirmed it's handled (or filed a follow-up) — never ack a signature you haven't actually investigated just to clear the list.

Guardrails

• ack_error/unack_error change state. Confirm the signature and rationale with the user before calling them unless they've already asked you to acknowledge a specific, named signature.
• Don't ack a signature solely because count_last_1h is 0 — a quiet signature can still represent an unfixed root cause that simply isn't triggering right now.
• Don't claim an incident is "new" or "unprecedented" without running similar_incidents first — recurring problems are common in this fleet and the whole point of this skill is catching that.
• When quoting sample_message/representative_messages/error_logs, keep quotes short and paraphrase the rest; don't dump full log bodies into the response.