| name | infra-triage |
| description | Infrastructure alert triage — dedup via YT search, deep PVE/K8s investigation, auto-escalation for recurring/flapping alerts, control plane deep dive for K8s controller nodes. |
| allowed-tools | Bash |
| user-invocable | true |
| metadata | {"openclaw":{"always":true}} |
Infrastructure Alert Triage (Level 1 + Level 2)
When you see a message starting with [LibreNMS] ALERT in #infra-nl-prod or #infra-gr-prod, you MUST automatically run this full triage flow using the exec tool. Do NOT ask questions — just execute.
Usage
./skills/infra-triage/infra-triage.sh <hostname> "<rule_name>" <severity> [--site nl|gr]
Site is auto-detected from hostname prefix (grskg* → GR, otherwise NL). Use --site to override.
Environment Variables
| Variable | Purpose |
|---|
FORCE_ESCALATE=true | Escalate regardless (set by n8n for flapping alerts) |
EXISTING_ISSUE=ID | Reuse this YT issue instead of creating new |
SKIP_ESCALATION=true | Skip escalation step (for burst/correlated triage) |
What the Script Does
Step 0: Issue Deduplication
Searches YouTrack for existing open issues with the same hostname (created within 24h). If found, reuses instead of creating a duplicate. Reopens if in "To Verify". Also searches for related issues (same alert rule on different hosts within 12h).
Step 1: Create/Reuse YouTrack Issue
Creates issue in the site's YT project (IFRNLLEI01PRD for NL, IFRGRSKG01PRD for GR) or reuses existing. Registers callback with n8n for dedup tracking. Sets custom fields (Hostname, Alert Rule, Severity, Alert Source, VMID, PVE Host).
Step 2: Investigation
- Classify device via LibreNMS API (OS, type, hardware)
- Find host in PVE (IaC repo grep for hostname)
- Check LXC/VM status on Proxmox via SSH
- Check connectivity (ping from container)
- K8s diagnostics if hostname matches k8s-*
- Control plane deep investigation for K8s controller nodes (etcd pods+logs, apiserver pods, CP resource usage)
- Check Docker services
- Query LibreNMS API for device status
Step 3: Post Findings to YouTrack
Posts investigation results as YT comment, including related issues and recurring alert flag.
Step 4: Escalation
Always escalates to Claude Code unless SKIP_ESCALATION=true. Includes escalation reason (standard/flapping/recurring).
CRITICAL RULES
- ALWAYS use the
exec tool to run the script. Do NOT describe what to do — execute it.
- Do NOT make changes. Level 2 is READ-ONLY. No
pct start, pct stop, docker restart, config edits, etc.
- Do NOT skip steps. Run ALL investigation steps even if the issue seems obvious.
- ALWAYS escalate to Claude Code after posting findings (unless SKIP_ESCALATION).
- After reporting output, add your CONFIDENCE score (see SOUL.md for format).
