create-alert-rule

# Skill: Nightingale (N9E) Alert Rule Creation Two tools: `import_alert_rule_template` (reuse the validated rules in integrations) and `create_alert_rule` (hand-write rules). **Prefer reusing the rules in integrations**: the rules under `integrations/<component>/alerts/` are finished products that have been hand-tuned and validated (severity, duration, evaluation interval, appended tags, annotations, and effective time windows are all configured), and their quality far exceeds hand-crafted rules. **But import as many as the user needs—do not blindly import the whole pack**: - The user wants "a set / common" alerts for some component → import the whole pack (do not pass `names`) - The user names only a few categories (e.g. "disk and memory alerts") → import only those (pass them in `names`) - The user wants a **single** alert for a specific metric → if the pack happens to contain a matching rule, import just that one (pass one entry in `names`) or borrow its expression; if the pack has nothing that fits, hand-write with `create_alert_rule` Only go to Approach B (`create_alert_rule`) when integrations has no matching rule, or when the user provides a fully custom query condition. When a component ships both a categraf and an exporter rule pack, **prefer categraf**: first probe whether the categraf metrics have data; if found, use the categraf pack, otherwise fall back to the exporter pack (see Approach A step 3 for details). | User need | What to do | |------|-----------| | Configure a set / a few alerts for a standard component (Linux / MySQL / Redis / Kafka / PostgreSQL / Elasticsearch / Ceph / Oracle / Windows / MongoDB …) | Approach A: `preview_alert_rule_template` to see what's in the pack → `import_alert_rule_template` to import as needed | | A single alert for a specific metric, and the integration pack has a matching rule | Approach A: `import_alert_rule_template` + `names` to import just that one | | Fully custom rules that integrations doesn't cover (including niche conditions like Loki / ES / SQL query values / log counts) | Approach B: `create_alert_rule` | > ⚠️ **Do NOT use this skill for bulk YAML imports**—when the user provides a URL or a YAML file, awesome-prometheus-alerts, node-exporter.yml, and the like, use import-prom-rule instead. ## Step 1 (common to both paths): determine the business group and data source - Call `list_busi_groups` to get the list of business groups - Call `list_datasources` to get the list of data sources, and find the data source ID of the matching type (rule packs are mostly the Prometheus type) - If the session context already has a preselected `busi_group_id` / `datasource_id`, use it directly—**do not** call `list_*` again - `cate=host` (host-unreachable class) does not need a data source ### Business group selection rules 1. If the user explicitly specified a business group name or ID, use it directly 2. Otherwise, by priority: a. **Prefer the business group with `is_default: true`** (usually "Default Busi Group" or a group whose name contains "Default") b. If there is only one business group, use it directly c. When there are multiple candidates and none is the default, **do not blindly take the first one**; list them in your reply and ask the user to confirm --- ## Approach A: reuse validated rules in integrations (as needed: single / batch / whole pack) `import_alert_rule_template` imports rules from the rule pack under `integrations/<component>/alerts/` (each rule's severity, duration, evaluation interval, appended tags, annotations, and effective time window are all preserved), binds the rules to the data source you select, and uniformly switches them to the enabled state. **How many rules are imported is controlled by the `names` parameter**, driven by the user's need. ### Steps 1. See which integration components are available: ``` list_files(base="integrations") ``` 2. See which rule pack files exist under that component: ``` list_files(base="integrations/Linux", path="alerts") ``` 3. **Pick the right rule pack file (prefer categraf)**. Rule packs often come in two collection styles, categraf and exporter, and the filename convention varies per component (for Linux it's `linux_by_categraf.json` / `linux_by_exporter.json`, but it may use other suffixes). **Prefer the pack whose filename contains `categraf`**: - First probe whether the categraf metrics have data in the environment: `list_metrics(datasource_id=<X>, keyword="<categraf metric keyword>")`. - Use a categraf-specific metric name that distinguishes the two styles as the keyword, e.g. for Linux use `cpu_usage` (categraf uses `cpu_usage_idle`, which node_exporter does not have), `mem_used_percent`, `disk_used_percent`; for Redis use `redis_used_memory`; for MySQL use `mysql_global_status_queries`. - **As long as the categraf metric returns data (a non-empty array), use the pack whose filename contains `categraf`** directly, without comparing against exporter. - Only fall back to the pack whose filename contains `exporter` (e.g. `linux_by_exporter.json`) when the categraf metric has no data, and use the corresponding exporter metric keyword (e.g. `node_cpu_seconds_total`) with `list_metrics` to confirm data exists before importing. 4. **See which rules are in the pack (only when you need to select by name)**. You can skip this step when the user wants to import the whole pack; when the user wants only a few of them, or a single alert for a specific metric, preview first to get the exact rule names: ``` preview_alert_rule_template(component="Linux", file="linux_by_categraf.json") ``` It returns each rule's `name` / `cate` / `severity` / an expression summary / whether it is disabled (a small payload). **Do not** use `read_file` to view a large rule pack (it will be truncated); use this tool. 5. **Import as needed**: - Whole pack (the user wants "a set / common" alerts)—do not pass `names`: ``` import_alert_rule_template(group_id=<business group>, component="Linux", file="linux_by_categraf.json", datasource_id=<data source>) ``` - Batch / single (the user names a few categories or a specific metric)—`names` carries the exact rule names (from preview): ``` import_alert_rule_template(group_id=<business group>, component="Linux", file="linux_by_categraf.json", datasource_id=<data source>, names="[\"Machine load - high memory, please pay attention - categraf\"]") ``` > **Do not** use `read_file` to read out the entire rule pack and then call `create_alert_rule` one rule at a time—the pack may be large and get truncated, and converting rules one by one also drops fields. To see what's in the pack use `preview_alert_rule_template`; to persist it use `import_alert_rule_template`. ### Notes - **Strongly recommend passing `datasource_id`**: passing it binds the imported rules to that data source; omitting it makes the rules match every data source of that type (an overly broad scope). The `host` heartbeat rules in the pack don't need a data source, and the tool skips binding for them automatically. - The rules in a rule pack are **disabled** by default in the template (it is a browsable catalog). When importing, this tool **switches them to enabled** by default (`disabled=0`) so the rules take effect immediately. If the user wants to "import first but leave them disabled, and enable them one by one themselves", pass `disabled=1`. - The names written in `names` must exactly match the `name` returned by `preview_alert_rule_template`; any that don't match are listed in the returned `not_found_names`—double-check and retry. - Duplicate rules (already existing in the business group) are **skipped automatically** (`status=skipped_duplicate`); this is not a failure, and **do not** re-import with `name_prefix` just because you see a skip. Only use `name_prefix`/`name_suffix` when the user explicitly wants the rule to coexist with the existing one (e.g. a comparison test). - The return includes three counts—`created` / `skipped` / `failed`—plus details for each rule; just summarize them in the output. - Imported rules **are not bound to any notify rule** by default (to avoid sending alerts by mistake). If binding is needed, after importing, prompt the user to associate it manually on the rule, or switch to Approach B to create rules one by one and pass `notify_rule_ids`. --- ## Approach B: hand-write a custom rule (when integrations has no matching rule) Use `create_alert_rule`. Supports all data source types: **Prometheus / Loki / Elasticsearch / OpenSearch / TDengine / ClickHouse / MySQL / PostgreSQL / Doris / VictoriaLogs / Host**. > Before getting here, confirm that integrations really has no matching rule: common metrics for standard components (CPU/memory/disk/connection count…) are mostly in the rule packs, so reuse rather than hand-craft when you can. Even when hand-crafting, you can first run `preview_alert_rule_template` to glance at whether there's a ready-made expression you can borrow. The tool offers two invocation modes: 1. **Prometheus simplified path** (most common)—pass `prom_ql` + `threshold` + `operator` directly, and the tool builds the rule automatically 2. **Generic path**—pass `cate` + `rule_config_json`; **first obtain the structure of `rule_config_json` by reading `datasources/<cate>.md` via `read_file`**, then fill in the actual values ### Mode 1: Prometheus simplified path The most common scenario. You only need to fill in PromQL, threshold, and operator. Internally the tool splices the threshold into `prom_ql` to generate a v1-format rule (the v2 editor in the OSS n9e FE is gated by `IS_PLUS`, so only v1 can be used). ```json { "group_id": 1, "name": "CPU usage too high", "datasource_id": 1, "prom_ql": "avg by (ident) (100 - cpu_usage_idle{cpu=\"cpu-total\"})", "operator": ">", "threshold": 80, "severity": 2, "note": "CPU usage exceeded 80% for 1 minute", "for_duration": 60 } ``` ### Mode 2: generic path (non-Prometheus) For types such as Loki / ES / MySQL / TDengine / ClickHouse / Doris / VictoriaLogs / Host, you need to pass `cate` and `rule_config_json`. **Key step**: first read the reference doc for that data source type to obtain the `rule_config` structure template: ``` read_file(base="create-alert-rule", path="datasources/<cate>.md") ``` Then turn the `rule_config` object in the doc into a JSON string and pass it to the `rule_config_json` parameter. Example (MySQL alert): ```json { "group_id": 1, "name": "MySQL too many failed orders", "cate": "mysql", "datasource_id": 4, "severity": 2, "rule_config_json": "{\"queries\":[{\"ref\":\"A\",\"sql\":\"SELECT count(*) AS value FROM orders WHERE created_at >= NOW() - INTERVAL 5 MINUTE AND status='failed'\",\"keys\":{\"valueKey\":\"value\",\"labelKey\":\"\"},\"interval\":60}],\"triggers\":[{\"mode\":1,\"exp\":\"$A.value > 10\",\"severity\":2,\"recover_config\":{\"judge_type\":1}}]}" } ``` **⚠️ Important generic rules (apply to all non-Prometheus types)**: 1. **The `interval` field must be the total number of seconds**—do not write `interval_unit`. When the frontend saves, it converts `value × unit → seconds`, and when reading it derives the display unit back from the seconds. - Last 1 minute → `"interval": 60` - Last 5 minutes → `"interval": 300` - Last 1 hour → `"interval": 3600` - **Do not** write `"interval": 5, "interval_unit": "min"`—the FE will display it as 5 seconds. - The tool has a defensive fallback: if you accidentally write `interval_unit` or a bare interval smaller than 60, it converts it to seconds automatically, but it's best to write it correctly to begin with. **⚠️ Important limitations for OSS n9e SQL-type data sources (MySQL/PGSQL/CK/Doris)**: 1. **`keys.valueKey` is required**—the alias of the numeric column in the SELECT statement, usually `"value"`. Missing it reports `valueKey is required`. 2. **`$from`/`$to`/`$__timeFilter` are not substituted**—the OSS `macros.Macro` is a no-op. You must use the data source's native time functions, e.g. MySQL uses `NOW() - INTERVAL 5 MINUTE`, PG uses `NOW() - INTERVAL '5 minutes'`, CK uses `now() - INTERVAL 5 MINUTE`. 3. **TDengine is the exception**—it has its own variable substitution logic, and `$from`/`$to`/`$interval` are usable. ## create_alert_rule parameter reference (Approach B) | Field | Required | Description | |---|---|---| | `group_id` | ✅ | Business group ID (from `list_busi_groups`) | | `name` | ✅ | Rule name (must be unique within the business group) | | `cate` | ❌ | Data source type (default `prometheus`). Options: `prometheus` / `loki` / `elasticsearch` / `opensearch` / `tdengine` / `ck` / `mysql` / `pgsql` / `doris` / `victorialogs` / `host` | | `prod` | ❌ | Product type. When omitted, derived automatically from `cate` | | `datasource_id` | Conditionally required | Data source ID. **Not needed when `cate=host`**; required for all others | | `rule_config_json` | Conditionally required | The complete `rule_config` JSON string. **Can be omitted when `cate=prometheus` (use the simplified path)**; required for other types | | `prom_ql` | Conditionally required | PromQL query (only for the `cate=prometheus` simplified path). **Write only the query, not the threshold** | | `threshold` | Conditionally required | Trigger threshold (only for the simplified path) | | `operator` | ❌ | Default `>`. Options `>` `>=` `<` `<=` `==` `!=` | | `severity` | ❌ | Default 2. 1=Critical, 2=Warning, 3=Info | | `note` | ❌ | Alert notification body | | `eval_interval` | ❌ | Evaluation interval (seconds), default 30 | | `for_duration` | ❌ | Duration (seconds), default 60 | | `append_tags` | ❌ | Appended tags, multiple separated by spaces | | `runbook_url` | ❌ | Runbook URL | | `notify_rule_ids` | ❌ | JSON list of associated notify rule IDs, e.g. `"[1,2]"` | ## Approach B detailed steps (create_alert_rule) > See "Step 1 (common to both paths)" above for selecting the business group and data source. Below are the steps specific to Approach B. ### B-1: identify the data source type (cate) Based on the user's alert need, decide which `cate` to use: | User-need keyword | cate | Trigger condition | |---|---|---| | "Host CPU/memory/disk", "Prometheus metric" | `prometheus` | Metric threshold | | "Machine unreachable", "node offline" | `host` | Heartbeat timeout | | "Application error log", "Loki log" | `loki` | Log count | | "ES log", "Elasticsearch aggregation" | `elasticsearch` | Log aggregation | | "OpenSearch log" | `opensearch` | Log aggregation (same as ES) | | "MySQL abnormal query result" | `mysql` | SQL query value | | "PostgreSQL abnormal query result" | `pgsql` | SQL query value | | "ClickHouse metric/log" | `ck` | SQL query value | | "Doris log" | `doris` | SQL query value | | "TDengine time-series data" | `tdengine` | SQL query value | | "VictoriaLogs log" | `victorialogs` | LogsQL query |