notify-channel-copilot

# Nightingale (n9e) Notify Channel Modification ## Scope (first determine which layer the user is changing) The Nightingale notification pipeline has three layers, each with different pain points: | Layer | Entity | Key files | Does this skill cover it | |---|---|---|---| | **Channel** Notify Channel | `notify_channel` table, `NotifyChannelConfig` | `models/notify_channel.go`, `alert/sender/provider/*.go` | **Yes** | | Message Template Notify Template | `notify_tpl` table | `models/notify_tpl.go` | No (use `generate-message-template`) | | Notify Rule | `notify_rule` table | `models/notify_rule.go` | No | **How to decide**: - The user's wording mentions "URL/Webhook address/request header/timeout/proxy/signature/secret key/AppID/AppSecret/CorpID/integration"—**channel layer**, stay in this skill. - The user's wording mentions "template/body/field/variable/rendering/title/card color"—**template layer**, switch to `generate-message-template`. - The user's wording mentions "who to send to/recipient/subscription/filter/label matching"—**rule layer**, out of scope for this skill. --- ## Data model `NotifyChannelConfig` `models/notify_channel.go`: ```go type NotifyChannelConfig struct { ID int64 Name string // display name Ident string // channel type (the key that routes to a provider, see table below) Description string Enable bool ParamConfig *NotifyParamConfig // user parameters: contact_key + custom params RequestType string // http | smtp | script | flashduty | pagerduty | jira | discord | jsm_alert | slackwebhook | mattermostwebhook RequestConfig *RequestConfig // pick the matching sub-struct by ident/request_type Weight int } ``` `RequestConfig` is a union: only one field is filled depending on the channel type: | Field | Applicable ident | |---|---| | `HTTPRequestConfig` | all channels that use a pure HTTP webhook (dingtalk, feishu, wecom, telegram, callback, …) | | `SMTPRequestConfig` | `email` | | `ScriptRequestConfig` | `script` | | `FlashDutyRequestConfig` | `flashduty` | | `PagerDutyRequestConfig` | `pagerduty` | | `DingtalkAppRequestConfig` | `dingtalkapp` (DingTalk app, currently not registered, see `provider/init.go`) | | `FeishuAppRequestConfig` | `feishuapp` | | `WecomAppRequestConfig` | `wecomapp` | | `JiraRequestConfig` | `jira` (native Jira issue channel, `request_type=jira`) | | `DiscordRequestConfig` | `discord` (native Discord channel, `request_type=discord`; optional, the webhook URL is a rule param) | | `JSMAlertRequestConfig` | `jsm_alert` (native JSM alert channel, `request_type=jsm_alert`; optional `api_url`, the integration key is a rule param) | | `SlackWebhookRequestConfig` / `MattermostWebhookRequestConfig` | `slackwebhook` / `mattermostwebhook` (native webhook channels; optional, the webhook URL is a rule param) | --- ## Built-in ident overview (`models/user.go` + `alert/sender/provider/init.go`) | Ident | Provider | Request type | One-liner | |---|---|---|---| | `dingtalk` | DingtalkProvider | HTTP | DingTalk group robot webhook | | `wecom` | WecomProvider | HTTP | WeCom group robot webhook | | `feishu` | simpleHTTPProvider | HTTP | Feishu group robot markdown (early) | | `feishucard` | FeishuCardProvider | HTTP | Feishu message card (supports color switching/@-mentions) | | `lark` | simpleHTTPProvider | HTTP | Lark (international Feishu) markdown | | `larkcard` | LarkCardProvider | HTTP | Lark card | | `feishuapp` | FeishuAppProvider | HTTP (App) | Feishu app robot (DM/group) | | `wecomapp` | WecomAppProvider | HTTP (App) | WeCom self-built app | | `telegram` | simpleHTTPProvider | HTTP | Telegram Bot | | `discord` (`request_type=discord`) | DiscordProvider | HTTP | Discord webhook (embed, forum posts) | | `slackwebhook` (`request_type=slackwebhook`) | SlackWebhookProvider | HTTP | Slack incoming webhook (attachment with severity color) | | `mattermostwebhook` (`request_type=mattermostwebhook`) | MattermostWebhookProvider | HTTP | Mattermost incoming webhook (attachment with severity color) | | `slackbot` / `mattermostbot` | simpleHTTPProvider | HTTP | legacy generic HTTP (Bot mode is not natively supported yet) | | `jira` (`request_type=jira`) | JiraProvider | HTTP | Jira issues: create on trigger, comment and close on recovery | | `jsm_alert` (`request_type=jsm_alert`) | JSMAlertProvider | HTTP | JSM Operations alerts: create on trigger, close on recovery | | `jira` / `discord` / `jsm_alert` / `slackwebhook` / `mattermostwebhook` (legacy `request_type=http`) | callback | HTTP | legacy hand-made webhook configs | | `email` | EmailProvider | SMTP | Email | | `tx-sms` | TencentSmsProvider | HTTP | Tencent Cloud SMS | | `tx-voice` | TencentVoiceProvider | HTTP | Tencent Cloud voice | | `ali-sms` | AliyunSmsProvider | HTTP | Alibaba Cloud SMS | | `ali-voice` | AliyunVoiceProvider | HTTP | Alibaba Cloud voice | | `pagerduty` | PagerDutyProvider | HTTP | PagerDuty | | `flashduty` | FlashDutyProvider | HTTP | Flashduty integration | | `script` | ScriptProvider | Script | shell/python script (compatible with the legacy notify.py) | | `callback` | CallbackProvider | HTTP | generic HTTP callback | **Registration mechanism**: `Resolve()` in `provider/registry.go`: first does an exact lookup by `Ident` → if not found, falls back by `RequestType` to a generic provider (`callback`/`script`/`email`/`flashduty`/`pagerduty`). So **a custom ident (e.g. `my-internal-webhook`) works as long as `request_type=http`, falling back to callback**. --- ## `HTTPRequestConfig` field details ```go type HTTPRequestConfig struct { URL string // full URL, may contain {{ ... }} template variables Method string // GET | POST | PUT Headers map[string]string // header values may also contain template variables Proxy string // like http://proxy:port, empty means direct connection Timeout int // milliseconds, default 10000 Concurrency int // concurrency, default 5 RetryTimes int // default 3 RetryInterval int // milliseconds, default 100 TLS *TLSConfig // {Enable, CertFile, KeyFile, CAFile, SkipVerify} Request RequestDetail // {Parameters: query string, Form, Body} } ``` **The Body field is a string**; inside it you reference the event using Go template syntax, and **the actual data fields follow the field dictionary from `generate-message-template`** (`$event.RuleName`, `$labels.ident`, `timeformat`, `unescaped`, etc.). The Body is rendered through `html/template`, so `<` and `&` get escaped—a JSON Body is usually unaffected, but **when writing HTML tags in the template you need `{{unescaped "<b>..."}}`**. **URL / Headers / Parameters are rendered through templates the same way** (`alert/sender/provider/http_common.go:113-142 replaceVariables`). Key details: - Only when it contains `{{` does it go through `html/template`; otherwise it is kept verbatim (`needsTemplateRendering` filters it first). - The context is shared with the Body—the 6 variables from the next section. Examples: - Route the URL by severity level: `http://bot/notify?level={{$event.Severity}}` - Inject auth into a header: `Authorization: Bearer {{$params.token}}` - Concatenate all recipients' phone numbers into the query: `?ats={{range $i,$s := $sendtos}}{{if $i}},{{end}}{{$s}}{{end}}` --- ## Template context — variables you can use All HTTP-type providers share `alert/sender/provider/http_common.go:SendHTTPRequest`. When rendering Body / URL / Headers / Parameters, these 6 variables are injected uniformly (line 34-44): | Variable | Source | Typical use | |---|---|---| | `$event` | `events[0]`, the first `AlertCurEvent` of this batch | `{{$event.RuleName}}` / `{{$event.Severity}}` / `{{$event.TriggerValue}}` | | `$events` | `[]*AlertCurEvent`, the whole batch | callback default template `{{ jsonMarshal $events }}` | | `$sendtos` | `[]string`, resolved from each recipient's `contact_info` by the channel's `ContactKey` | `{{range $sendtos}}...{{end}}`, `{{ jsonMarshal $sendtos }}` | | `$sendto` | `sendtos[0]`, present only when `len(sendtos)>0` | single-recipient rendering | | `$params` | the channel's `ParamConfig.Custom.Params` | SMS template's `SignName` / `TemplateId` | | `$tpl` | custom fields attached in the notify template | used when decoupling the template from the channel | In addition, `$event.NotifyUsersObj` is also valid: a `[]*User` array containing the complete User objects of all recipients in this batch (`models/alert_cur_event.go:65` + `alert/dispatch/dispatch.go:971`). It is a field left over from the v6 notify.py era (`gorm:"-"`, runtime-only), and is still populated in v8. The difference from `$sendtos`: - `$sendtos`: a plain string array, already resolved by ContactKey, **preferred for simple scenarios**. - `$event.NotifyUsersObj`: a full User object array, used when the template needs multiple fields like Phone + Email + Username at the same time, e.g. `{{range $event.NotifyUsersObj}}{{.Phone}} {{.Username}}{{end}}`. **Where `$sendtos` comes from** (`alert/dispatch/dispatch.go:451-541 GetNotifyConfigParams`): Take the `user_ids` + `user_group_ids` from this notify_rule's NotifyConfig → query UserCache → read the value from each user's `contact_info` JSON by the channel's `ParamConfig.UserInfo.ContactKey` → deduplicate → build `[]string`. ⚠️ Therefore: - `ContactKey=phone` but the user's `contact_info.phone` is empty → this person doesn't make it into sendtos. - For a custom key like `ContactKey=dingtalk_robot_token`, the user's `contact_info` JSON must also have a key of the same name. Otherwise `$sendtos` is an empty array, and `$sendto` is simply undefined. - This is the most common root cause of "test notification works but real alerts don't go out" (see the dedicated "Test works but real alerts don't go out" section in the debugging chapter). --- ## The "required-fields map" for each channel Below is the minimal usable configuration for each channel—when the user asks "how do I integrate X", just tell them to fill in these few fields. ### 1) DingTalk group robot `dingtalk` - `request_type=http`, `HTTPRequestConfig`: - `URL`: the group robot webhook (DingTalk admin → group settings → Group Assistant → Add Robot → Custom) - `Method`: `POST` - `Headers`: `Content-Type: application/json` - `Body`: JSON markdown, see `provider/dingtalk_provider.go`. The `text` field must contain a **keyword** (DingTalk group robot allowlist check: the keyword you set when creating the robot in DingTalk must appear in the message text). - **Signing vs keyword**: the built-in DingTalk provider in n9e **only supports keyword validation**. The signing mechanism requires appending `&timestamp=&sign=` to the URL query, and there is currently no native toggle for it. If the user must use signing: 1. Use the `callback` channel and assemble the signed URL yourself (writing it in `URL` for render-time is awkward and needs a pre-script), or 2. Switch to the `script` channel to call a script. 3. **Recommended wording**: "For the DingTalk robot, please switch to keyword validation; the signing mechanism is not natively supported in the open-source version." - **@-mentions**: depend on the `at.atMobiles` / `atUserIds` arrays in the message body. The built-in DingTalk provider's request body is assembled by the message template, so **in the template use `{{batchContactsAts}}` or range over `$event.NotifyUsersObj` yourself to get `.Phone`**. ### 2) WeCom group robot `wecom` - `URL`: the group robot webhook - `Method`: `POST`, `Content-Type: application/json` - **Limitations**: the group robot markdown does not support `<font color>`; @-mentions rely on `mentioned_mobile_list`/`mentioned_list`; **the new version of WeCom has retired group robots**, so newly created groups no longer have a webhook address—at that point you can only switch to the `wecomapp` self-built app. ### 3) WeCom self-built app `wecomapp` - `request_type=http`, but uses `WecomAppRequestConfig`: - `CorpID`, `CorpSecret`, `AgentID` (obtained from the WeCom admin console) - `Proxy` / `Timeout` / `RetryTimes` / `RetrySleep` - Uses `provider/wecomapp_provider.go`, which automatically manages access_token refresh. - **Recipient field**: the user's `contact_info.wecom_userid` (filled in user management). ### 4) Feishu group robot (markdown) `feishu` - Uses `simpleHTTPProvider`, template-driven. Set `HTTPRequestConfig.URL` to the group robot webhook, and `Body` to a JSON template. - **Signature validation**: a Feishu group robot secret, like DingTalk signing, falls under "timestamp + signature embedded in the message body". **The built-in feishu provider in n9e likewise has no automatic signing.** Solutions: 1. When creating the group robot, **do not check "Signature validation"**, and use "Custom keyword" or "IP allowlist" instead. 2. If signing is mandatory → use a custom `script` channel. - **Backslash Bad Request 9499**: the Feishu webhook receives JSON, and a backslash `\` is a JSON escape character. Label values like the Windows path `D:\foo` or `device="D:"` will break the JSON if they go directly into the body. **In the template use `{{$labels.path | jsonMarshal}}` to convert the string into a valid JSON string** (with quotes), or use `label_replace` at the PromQL/rule source to strip out the `\`. ### 5) Feishu card `feishucard` / Lark card `larkcard` - Uses `FeishuCardProvider` / `LarkCardProvider`, sending Feishu v2 message cards. - Configuration items are the same as `feishu` (URL + optional secret). `Body` is a complete `interactive` card JSON. - **Card color switching**: a Feishu card only recognizes enumerated colors (`red / orange / yellow / green / turquoise / blue / indigo / purple / carmine / grey`); writing hex is invalid. The color is written in the `header.template` field and rendered by the template based on `IsRecovered / Severity`. - **@-mentions**: use `<at email=...></at>` or `<at id=...></at>`; the template uses `{{batchContactsAtsInFeishuEmail $event.NotifyUsersObj}}` or `{{batchContactsAtsInFeishuId ...}}`. ### 6) Feishu self-built app `feishuapp` - Uses `FeishuAppRequestConfig`: