modify-task-tpl

# Nightingale (n9e) Alert Self-Healing Script (task_tpl) Generation Nightingale alert self-healing is the **ibex subsystem**: when an alert rule's `callbacks` field is written as `${ibex}/<task_tpl_id>`, after an alert event fires it pulls up the script of the corresponding task_tpl on the machine matching the current event's `TargetIdent` and runs it on that machine's categraf. The script receives the labels of the current alert through **stdin**. This skill focuses on **writing/modifying the `task_tpl.script` field itself**—it does not cover creating alert rules, configuring recipients, or editing notification templates. --- ## 1. Scope: First Determine Which Layer the User Is Changing The Nightingale alert pipeline has four layers, and each goes through a different skill: | Layer | Entity | Key files | Handled by this skill | |---|---|---|---| | **Self-healing script** task_tpl | `task_tpl` table | `models/task_tpl.go`, `alert/sender/ibex.go` | **Yes** | | Alert rule | `alert_rule` table | `models/alert_rule.go` | No (use `create-alert-rule`) | | Notification template | `notify_tpl` table | `models/notify_tpl.go` | No (use `generate-message-template`) | | Notification channel | `notify_channel` table | `models/notify_channel.go` | No (use `notify-channel-copilot`) | **Judgment criteria**: the user's wording contains "script/shell/bash/python/jq/parse/execute/timeout"—this skill; contains "PromQL/threshold/trigger condition"—alert rule; contains "template/body/field rendering"—message template; contains "URL/Webhook/signature"—notification channel. --- ## 2. Data Model (What the User Can Fill In, Which Fields Nightingale Actually Uses) ### `TaskTpl` (`models/task_tpl.go:17-35`) | Field | Type | Meaning | |---|---|---| | `id` | int64 | Primary key; this is what `${ibex}/<id>` in the alert rule callbacks references | | `group_id` | int64 | Business group (permission boundary, validated by `CanDoIbex`) | | `title` | string | Template name; at execution time it is concatenated into `<title> FH: <hostname>` and written as the task title | | `script` | string | **The main field this skill operates on** | | `args` | string | Command-line arguments. If the caller does not pass `args` when the alert fires and executes, the default value here is used | | `tags` | string | Space-separated tags, used for template list filtering/classification, does not affect execution | | `account` | string | Which user identity to run as on the target machine (e.g. `root`) | | `batch` | int | Number of hosts running concurrently per batch. **Self-healing usually only runs on the host that triggered it, so 0 is fine** | | `tolerance` | int | Allowed failure count within a batch. Leave at 0 for single-host self-healing scenarios | | `timeout` | int | **Seconds**. 0 → defaults to 30; > 5 days → rejected | | `pause` | string | Pause schedule between batches (cron style). Basically unused for self-healing | ### `TaskForm` (`models/task_tpl.go:351-365`) — the Carrier Actually Dispatched Alert-triggered self-healing goes through `alert/sender/ibex.go CallIbex` → constructs a `TaskForm` → `TaskAdd` writes to ibex: - `AlertTriggered: true` —— edge data centers go through Redis direct-dispatch to categraf, not relying on the central DB (`ibex.go:244-276`) - `Stdin: <JSON string>` —— **what this skill cares about most**, explained in detail in the next section - `Hosts: []string{event.TargetIdent}` —— executes only on the machine the alert event matched - `Title: tpl.Title + " FH: " + host` —— easy to recognize in execution history (FH = From Host) ### CleanFields Hard Constraints (`models/task_tpl.go:137-189`) These validations run before saving or dispatching; violations are rejected outright: - `timeout == 0` → automatically set to 30 - `timeout > 3600*24*5` → error "longer than five days" - `title` / `args` / `pause` / `tags` containing `str.Dangerous` characters (`` ` ``, `$()`, `&&`, etc.) → error - `script` automatically `\r\n → \n` (resolves the CRLF problem from Windows editor copies) - `script == ""` → error "script is required" --- ## 3. ⚠️ The Truth About stdin (the Easiest Pitfall, **Must Read**) ### 3.1 stdin Is a Flat JSON of `map[string]string`, **Not a `$event` Object** Many users copy syntax like `{{$event.RuleName}}` from the message template docs—it **does not work at all**. `$event` is the context of the `notify_tpl` template; there is no template rendering process inside an ibex self-healing script—what it receives is just a plain JSON string. **The construction logic of stdin** (`alert/sender/ibex.go:118-142`): ```go tagsMap := make(map[string]string) for _, pair := range event.TagsJSON { // flatten event labels one by one k, v := splitOnce(pair, "=") tagsMap[k] = v } // inject 3 built-in keys tagsMap["alert_severity"] = strconv.Itoa(event.Severity) tagsMap["alert_trigger_value"] = event.TriggerValue tagsMap["is_recovered"] = strconv.FormatBool(event.IsRecovered) tags, _ := json.Marshal(tagsMap) // serialize the whole thing into a string in.Stdin = string(tags) ``` **Example of a real stdin payload**: ```json { "ident": "host01", "instance": "host01:9100", "job": "categraf", "alert_severity": "2", "alert_trigger_value": "92.5", "is_recovered": "false" } ``` Note: all values are strings (`alert_severity` is `"2"`, not `2`); to parse them into numbers in the script you have to convert them yourself. ### 3.2 Three Fields You "Want but Can't Get", and Their Solutions | Field you want | Why you can't get it | Solution | |---|---|---| | Friendly names like `rule_name` / `rule_id` / `trigger_time` / `severity` | stdin only carries labels, not event metadata | In PromQL use `label_replace(..., "rule_name", "...", "...", "...")` to inject the info into a label; or switch to a callback medium (HTTP carrying the whole event JSON) | | Recipient / notification group | Self-healing and notification rules are **two independent channels**; self_heal only looks at `alert_rule.callbacks`, not `notify_rule` | If you want recipients → go through the callback medium; stdin is not this path | | Labels aggregated away by PromQL (e.g. k8s `namespace`/`deployment`) | `sum(...)` without `by(...)` drops labels | **The PromQL in the alert rule must be written as `sum by (instance, namespace, deployment)(...)`**; only the retained labels will enter stdin | ### 3.3 Templates for Reading stdin in Three Languages (Must Include When Generating Scripts) **shell + jq (most common)**: ```bash #!/bin/bash set -euo pipefail PAYLOAD=$(cat) # read all of stdin at once to avoid blocking [ -z "$PAYLOAD" ] && { echo "FATAL: empty stdin payload"; exit 2; } IDENT=$(echo "$PAYLOAD" | jq -r '.ident // empty') SEV=$(echo "$PAYLOAD" | jq -r '.alert_severity // "3"') VAL=$(echo "$PAYLOAD" | jq -r '.alert_trigger_value // "0"') [ -z "$IDENT" ] && { echo "FATAL: no ident in stdin"; exit 2; } echo "[$(date -Iseconds)] ident=$IDENT severity=$SEV value=$VAL" ``` **shell fallback without jq** (use grep+sed, suitable for minimal containers): ```bash PAYLOAD=$(cat) IDENT=$(echo "$PAYLOAD" | grep -oE '"ident"[[:space:]]*:[[:space:]]*"[^"]*"' | sed 's/.*"\([^"]*\)"$/\1/') ``` **python** (when the processing logic is complex): ```python #!/usr/bin/env python3 import sys, json, time try: data = json.load(sys.stdin) except json.JSONDecodeError as e: print(f"FATAL: bad stdin json: {e}", file=sys.stderr) sys.exit(2) ident = data.get("ident", "") sev = int(data.get("alert_severity", "3")) val = float(data.get("alert_trigger_value", "0")) if not ident: print("FATAL: no ident", file=sys.stderr) sys.exit(2) print(f"[{time.strftime('%FT%T')}] ident={ident} sev={sev} val={val:.2f}") ``` **go** (rarely used, minimal skeleton): ```go package main import ( "encoding/json" "fmt" "os" ) func main() { var d map[string]string if err := json.NewDecoder(os.Stdin).Decode(&d); err != nil { fmt.Fprintln(os.Stderr, "FATAL:", err) os.Exit(2) } fmt.Println("ident:", d["ident"]) } ``` --- ## 4. ⚠️ The Truth About `is_recovered` (**Don't Hard-Code Dead Code**) **ibex self-healing does not fire on recovery events**—`alert/sender/ibex.go:39-42`: ```go func (c *IbexCallBacker) CallBack(ctx CallBackContext) { ... if event.IsRecovered { logger.Infof("event_callback_ibex: event is recovered, event: %s", event.Hash) return // ← returns directly, never calls handleIbex at all } ... } ``` **Therefore `is_recovered` in stdin is always the string `"false"`.** Writing `if [ "$IS_RECOVERED" = "true" ]; then ...` is dead code. **The correct path**: - If you want to "also run an action on alert recovery (send email / close ticket / notify IM)" → use **notify_rule + callback medium**, check "also notify on recovery" in the notify_rule, and process the event JSON after the callback URL receives it. - If you want to "trigger another script on a recovery event" → same as above; **do not** try to reuse task_tpl. A core pain point is "I also want it to trigger on recovery", and the common idea of "adding an is_recovered check" is actually a non-solution—fundamentally you need an architectural change to ibex so it accepts recovery events, or the user must switch to the callback channel. --- ## 5. timeout / batch / tolerance / pause —— Semantics of the Numeric Fields ### 5.1 timeout (seconds) - Defaults to 30 seconds (auto-filled by `CleanFields`) - Maximum of 5 days (anything longer errors out; by design self-healing is a short-cycle task) - After timeout **the process is SIGKILLed**; stdout/stderr is written to the task_host table before being killed - **No direct relationship to the alert evaluation interval**—a 60s evaluation cycle does not mean the script only has 60s to run **Typical value reference**: | Scenario | Suggested timeout | |---|---| | reload config / systemctl restart | 30 ~ 60 | | clean logs / image cache (several GB) | 120 ~ 300 | | jstack/jmap heap dump | 300 ~ 600 | | yum/apt package install | 600 ~ 1800 | ### 5.2 Distinguish Three "Timeout-Class" Root Causes | Symptom | Real root cause | Solution | |---|---|---| | docker-compose self-healing script times out | mistakenly treating telegraf as the ibex-agent, **the channel was never established at all** | install categraf (not telegraf), check the `categraf -test` output | | notification script with timeout=0 is killed immediately | fe form DefaultTimeout=0 BUG | already fixed in backend CleanFields (0 → 30) | | self-healing script execution times out | the script itself is slow (remote yum / large file copy) | increase `timeout`; consider using a p2p tool to transfer files (do not use ibex as a file distributor) | **Do not lump them together**—when the user says "timeout", first ask about the scenario. ### 5.3 batch / tolerance / pause - **batch**: number of hosts running concurrently per batch. Single-host self-healing triggered by an alert doesn't need it; leave at 0. Only manual batch tasks (ops scenarios) use it. - **tolerance**: allowed failure count within a batch. Same as batch, leave at 0 for self-healing scenarios. - **pause**: pause between batches (e.g. `00:00-08:00,17:00-23:59` means pause during these time ranges). Basically not used for self-healing—an alert can fire at any time, so you can't use pause to restrict working hours. **If you really want to restrict working hours**, do it in the alert_rule's `enable_in_bg` / `enable_stime` fields, or use the notify_rule's time window. --- ## 6. Dangerous Command List (Must Be Avoided or Guarded When Generating) ### 6.1 Blacklist (**Refuse to Generate Outright**, Even If the User Explicitly Asks) | Command | Risk | |---|---| | `rm -rf /` / `rm -rf /*` / `rm -rf $UNSET_VAR/` | Wipes the entire disk | | `mkfs.*`, `dd of=/dev/sda`, `shred /dev/sda` | Filesystem destruction | | `shutdown`, `reboot`, `init 0`, `init 6`, `halt`, `poweroff` | Whole-machine shutdown—a self-healing script should not have this power | | `iptables -F` / `ufw disable` / `firewall-cmd --reload` without backup | Network/security policy loss | | `chmod -R 777 /`, `chown -R nobody:nobody /` | Permission destruction |