infra-triage

name	infra-triage
description	Infrastructure alert triage — dedup via YT search, deep PVE/K8s investigation, auto-escalation for recurring/flapping alerts, control plane deep dive for K8s controller nodes.
allowed-tools	Bash
user-invocable	true
metadata	{"openclaw":{"always":true}}

Infrastructure Alert Triage (Level 1 + Level 2)

When you see a message starting with [LibreNMS] ALERT in #infra-nl-prod or #infra-gr-prod, you MUST automatically run this full triage flow using the exec tool. Do NOT ask questions — just execute.

Usage

./skills/infra-triage/infra-triage.sh <hostname> "<rule_name>" <severity> [--site nl|gr]

Site is auto-detected from hostname prefix (grskg* → GR, otherwise NL). Use --site to override.

Environment Variables

Variable	Purpose
FORCE_ESCALATE=true	Escalate regardless (set by n8n for flapping alerts)
EXISTING_ISSUE=ID	Reuse this YT issue instead of creating new
SKIP_ESCALATION=true	Skip escalation step (for burst/correlated triage)

What the Script Does

Step 0: Issue Deduplication

Searches YouTrack for existing open issues with the same hostname (created within 24h). If found, reuses instead of creating a duplicate. Reopens if in "To Verify". Also searches for related issues (same alert rule on different hosts within 12h).

Step 1: Create/Reuse YouTrack Issue

Creates issue in the site's YT project (IFRNLLEI01PRD for NL, IFRGRSKG01PRD for GR) or reuses existing. Registers callback with n8n for dedup tracking. Sets custom fields (Hostname, Alert Rule, Severity, Alert Source, VMID, PVE Host).

Step 2: Investigation

• Classify device via LibreNMS API (OS, type, hardware)
• Find host in PVE (IaC repo grep for hostname)
• Check LXC/VM status on Proxmox via SSH
• Check connectivity (ping from container)
• K8s diagnostics if hostname matches k8s-*
• Control plane deep investigation for K8s controller nodes (etcd pods+logs, apiserver pods, CP resource usage)
• Check Docker services
• Query LibreNMS API for device status

Step 3: Post Findings to YouTrack

Posts investigation results as YT comment, including related issues and recurring alert flag.

Step 4: Escalation

Always escalates to Claude Code unless SKIP_ESCALATION=true. Includes escalation reason (standard/flapping/recurring).

CRITICAL RULES

1. ALWAYS use the exec tool to run the script. Do NOT describe what to do — execute it.
2. Do NOT make changes. Level 2 is READ-ONLY. No pct start, pct stop, docker restart, config edits, etc.
3. Do NOT skip steps. Run ALL investigation steps even if the issue seems obvious.
4. ALWAYS escalate to Claude Code after posting findings (unless SKIP_ESCALATION).
5. After reporting output, add your CONFIDENCE score (see SOUL.md for format).