| name | notification-design |
| description | EEMUA 191 design gate for alarms and notifications. Run this BEFORE creating or modifying ANY alert/notification anywhere in the ecosystem — a PrometheusRule, a Gatus alert, a Home Assistant / Node-RED automation that notifies, an Inovelli LED-bar notification, an app webhook (Pushover/ntfy), a Flux alert. Walks the rationalization questions that decide whether the alert should exist, what severity it gets, and which channel (page / push / TTS / LED bar / dashboard) delivers it. Also use when fighting alert fatigue, floods, or auditing existing alerts. |
| allowed-tools | Read, Edit, Write, Bash, Grep, Glob |
Notification design gate (EEMUA 191)
EEMUA Publication 191 (with ISA-18.2 / IEC 62682) is the industrial alarm-management standard.
Its core insight scales to a homelab: a notification is a demand on a human's attention, and
attention is a fixed budget. Here the "operator console" is Derek's phone, there is ONE
operator, and he is asleep a third of the day.
This skill is a gate, not a reference. Whenever an alert/notification is about to be born —
regardless of which system delivers it — walk the steps below IN ORDER. Most proposed alerts
should die at Step 1 or get demoted at Step 2. That is the skill working, not a failure.
The prime directive: an alarm requires an operator ACTION. If there is no action, it is not
an alarm — it's an event. Events go to dashboards/logs (Grafana, VictoriaLogs, HA logbook),
not to a phone.
Step 1 — Rationalize: does this deserve to exist?
Answer all five in writing (they become the alert's annotations/message):
- Cause — what happened?
- Consequence — what breaks if nobody acts?
- Response — what does the person seeing it actually DO?
- Time to respond — how long before the consequence lands?
- Audience — WHO can act on it? This decides which console it's even allowed on:
- Derek-only (anything cluster/infra: etcd, VolSync, certs, Flux…) → phone channels
(ntfy/Pushover) ONLY. Never the LEDs or TTS.
- Household (Derek + Sara: trash, dryer, doors, doorbell, alarm, leaks) → LED/TTS
eligible, and often the LED is the BETTER channel — Sara can't see Derek's phone.
Kill criteria — if any of these hit, do NOT create the notification (on this channel):
- No response exists ("just good to know") → dashboard it — or, for home state worth a
glance, demote to an Inovelli LED slot (see channel ladder below). Ambient light costs
~zero attention; a push costs an interruption.
- It self-heals (pod restart, retry, HPA, reconnect) → the system doing its job is an event.
Alert only if it's still broken after self-healing had time to work.
- The response is automatable → automate it; alert only on automation failure.
- An existing alert already covers the consequence (check stock kube-prometheus-stack rules
and
kubectl get prometheusrules -A) → don't create a twin.
- It's a symptom of another alert (node down → everything on it) → add an inhibition rule
instead of a new alert.
Step 2 — Prioritize: consequence × urgency, then pick the channel to match
Priority = consequence severity × time available to respond. Not how important the component
feels, not how proud we are of the app.
Channel intrusiveness ladder — pick the LOWEST rung that still gets the response in time.
Every rung you climb spends more of the attention budget:
| Rung | Channel | Costs the human | Use for |
|---|
| 0 | Dashboard / logs | nothing until they look | events, history, debugging |
| 1 | Inovelli LED slot (per-LED, persistent) | a glance at any switch | standing home state: trash out?, door open, dryer done, alarm armed |
| 2 | Inovelli full-bar effect (transient) | ambient attention in the room | local, now-ish events: doorbell, timer expiry, escalations |
| 3 | ntfy push | an interruption, follows you | act today-ish, or away-from-home relevance |
| 4 | TTS announce / Pushover page (bypasses DND) | wakes / stops everyone | act NOW: safety, security, data loss |
Two channels can pair (alarm armed = LED slot + arming push), but each channel must
independently pass the gate — "also send a push" is a decision, not a default.
Consoles have audiences (EEMUA: route alarms to the operator who can respond). Rungs 0 and
3–4 are Derek's console. Rungs 1–2 (LEDs) and TTS are the HOUSEHOLD console — Sara sees them
too. A notification on a shared surface must be actionable by whoever is looking at it: "trash
night" passes; "etcd flapping" is pure noise to Sara and erodes the trust that makes the trash
light work. Infra never lights the panel. Rare, deliberate exceptions only where the household
response is real (e.g. "internet down" = "it's known, don't power-cycle the router") — chosen
case by case, never defaulted.
| severity | Delivery (cluster) | Test | EEMUA share |
|---|
critical | Pushover page + ntfy urgent (bypasses DND) | "Would I want to be woken at 3am for this?" Data loss, safety, security, power, whole-cluster. | ~5% |
warning | ntfy high | Act today-ish: degraded, trending toward full, backup missed. | ~15% |
info | ntfy low / mostly null-routed | No action. Prefer no notification at all. | ~80% |
EEMUA's target distribution is ~80% low / 15% medium / 5% high. The repo's custom rules
currently sit near 50/50 critical/warning — bias DOWN. Media-stack anything is almost never
critical (Plex down at 3am costs nothing; make it a warning handled at breakfast). Legitimately
critical: UPS/power, storage/data-loss risk (VolSync, CNPG, Ceph health), security (cert expiry
imminent, Authentik down), Flux fully broken.
Smart-home ladder, same logic: critical = water leak, smoke/CO, freezer warm, garage open late
(bypass DND / TTS announce); high = door unlocked at bedtime, washer done, important battery
low; motion/lights/presence = events, no push.
Step 3 — Debounce: design out the noise before it ships
- PrometheusRule:
for: on everything user-facing — ≥ 2× scrape interval AND ≥ the time the
condition self-heals in practice (pod restart ≈ 5m). Prefer predict_linear ("disk full in
4h") over static thresholds; if static, pick a value the metric doesn't flap around.
- Gatus: the defaults
failure-threshold: 5 / success-threshold: 3 (in
kubernetes/apps/observability/gatus/app/resources/config.yaml) are the debounce — don't
lower them per-endpoint without a reason.
- Home Assistant:
for: on triggers, condition guards for state-based suppression (don't
announce "door open" while actively unloading groceries).
- Direct webhooks (*arr, sabnzbd, custom scripts): events silent/low priority
(sabnzbd uses Pushover -2 — correct), action-required high (*arr bumps to 1 only for
ManualInteractionRequired — correct). Never page priority for something with no response.
Step 4 — Route: use the existing pipeline, don't invent a channel
Prefer PrometheusRule → Alertmanager over bespoke webhooks: central grouping, inhibition,
silencing, and history beat N pipelines. The existing plumbing:
- Routing:
kubernetes/apps/observability/kube-prometheus-stack/app/alertmanagerconfig.yaml —
default receiver ntfy; severity=critical → Pushover (continue: true) → ntfy. Groups by
[alertname, job], groupWait 1m, groupInterval 10m, repeat 12h. Critical inhibits
same-alertname+namespace warnings. Unactionable stock alerts go to the null receiver
(InfoInhibitor, Watchdog, CPUThrottlingHigh, …) — extend that list rather than tolerate noise.
- ntfy topics/priorities:
kubernetes/apps/observability/alertmanager-ntfy/app/resources/config.yml
— media namespace → topic plex, else alertmanager; critical=urgent, warning=high, info=low.
- Placement: the app's own
.../app/prometheusrule.yaml, listed in its kustomization.yaml.
- HA (out-of-band): route notify → ntfy topics for parity; critical may use ntfy urgent or Pushover.
- Temporary suppression = silence-operator
Silence CRs
(kubernetes/apps/observability/silence-operator/silences/silences.yaml), always with a
comment saying why. Shelving is triage, not a fix — a months-old silence means the alert
should be rewritten or deleted.
Inovelli LED platform (rungs 1–2) — the house annunciator panel
Full design + hardware grammar live in the second brain:
~/second-brain/Projects/HomeLab/Inovelli LED Notification Platform.md (30 VZM31-SN switches,
slot assignments, zone map, MQTT payloads). Read it before wiring anything. Design rules:
- The panel is the household's, not the homelab's. LED notifications must pass the Step 1
audience test for Sara, not just Derek. No cluster/infra status on the switches — the panel's
entire value is that every light on it means something a household member should do.
- All LED writes go through the Node-RED
led-notify dispatcher (Phase 1 shipped). Producers
emit the normalized message (id, action: set|clear, slot/full-bar, zone, color,
effect, level, priority) — never publish to zigbee2mqtt/<switch>/set directly. The
dispatcher owns zone fan-out, priority preemption, Night-mode dimming, and state re-assert.
- Per-LED slot = standing status; full-bar = transient event. Don't put a transient on a
slot or a standing state on the full bar. Slots are the EEMUA "annunciator": same LED means
the same thing on every switch in the house — check the slot table before claiming one, and
update the note when you do.
- Honor the color language (red security, orange physical-action-needed, green complete,
blue people, yellow reminder, purple fun). EEMUA is emphatic about consistent presentation:
a color that means different things in different rooms is worse than no light.
- Clear discipline is the standing-alarm rule made physical. Everything sent with
duration: 255 needs a named owner and a defined clear condition; an LED that stays on after
the condition cleared trains the household to ignore the panel — same failure mode as a
stale Alertmanager alert. Prefer auto-expiring durations for transients.
- Zones are state-based suppression: bedtime-relevant only upstairs, guest wing quiet unless
guest mode, Night mode caps brightness. A new notification must declare its zone, not default
to house-wide.
Step 5 — Write the message: cause → consequence → response on a lock screen
annotations:
summary: "UPS on battery — {{ $value }} min runtime left"
description: >-
Mains power lost. Cluster shuts down cleanly at 5 min runtime.
Check breaker/utility; shed load if outage will be long.
The alertmanager-ntfy bridge already adds severity emoji, namespace tags, click-through, and
action buttons (Prometheus / Grafana alert history / pre-filled silence) — don't duplicate them.
Keep wording consistent per class of problem so pattern recognition works half-asleep.
Step 6 — Watch it for a week
First-week flapping = tune NOW, not later. Attention budgets (EEMUA rates, one-operator scale):
- Pushover pages: ~0/week steady state. Even 1/day desensitizes; the channel dies when it matters.
- ntfy high: a handful/day, grouped.
- Any alert firing >5×/week is a "bad actor" — EEMUA: the top 10 bad actors typically cause
>50% of total load; fixing them is the highest-leverage alerting work there is.
- Standing alerts are debt — firing-for-days trains flood-blindness. Fix or shelve (visibly).
The LED equivalent: the nightly orphan watchdog should never be the thing that clears your
notification — if it is, your clear condition is broken.
- A page that turned out non-actionable is a defect: demote or fix it the same day. Same for
an LED slot nobody glances at anymore — demote to dashboard or delete.
The pipeline is itself a system to alarm on
EEMUA devotes a whole chapter to alarm-system health: a dead notification path fails SILENT —
no alerts looks exactly like all-healthy. Design rules:
- Channel-failure alerts must cross channels. An "ntfy send failing" alert routed to ntfy
tells nobody anything. Send-failure alerts for channel A go to channel B (and vice versa).
- Dead man's switch. The stock
Watchdog alert exists to be an always-firing heartbeat
consumed by something OUTSIDE the cluster (healthchecks.io or similar) that pages when the
heartbeat STOPS. Null-routing it means end-to-end pipeline death goes unnoticed.
- Volume is a channel-health input, not just an annoyance. ntfy rate-limits per visitor;
a noisy day → 429s → Alertmanager retries (up to ~20×) → MORE requests → sustained saturation
where real alerts get dropped after retry exhaustion. Floods don't just distract the human,
they break the pipe. Watch
alertmanager_notifications_failed_total.
- Test the paging path on a schedule. A monthly deliberate test page (and LED test pattern)
is the fire drill — the worst time to learn Pushover credentials expired is during an outage.
Auditing the existing posture
grep -rh "severity:" kubernetes/apps/*/*/app/*prometheusrule*.yaml kubernetes/apps/*/*/*/prometheusrule.yaml 2>/dev/null | sort | uniq -c
kubectl -n observability exec alertmanager-kube-prometheus-stack-0 -- amtool alert query --alertmanager.url=http://localhost:9093
Related
- Docs pipeline overview:
docs/src/apps/observability.md (Mermaid flow)
- LED platform design (slots, zones, dispatcher, MQTT grammar):
~/second-brain/Projects/HomeLab/Inovelli LED Notification Platform.md
- Skills:
gatus-monitoring (uptime checks), externalsecrets (pushover/ntfy tokens)
- Source: EEMUA Publication 191 (4th ed., 2024) — https://www.eemua.org/products/publications/digital/eemua-publication-191 ; aligned with ISA-18.2 / IEC 62682
