self-monitoring-resource-alarms

name	self-monitoring-resource-alarms
description	Design and implement a background self-monitor that watches a service's OWN resource footprint (its data-store bytes, WAL bytes, per-table row counts, and growth rate between samples) and raises graduated, dedup-governed warn/crit alarms before a runaway fills the disk. Use when a daemon or service can silently bloat its own storage, when adding self-observability, resource alarms, growth-rate detection, per-tenant footprint metering, or global unhandledRejection/uncaughtException failure-visibility handlers, or when auditing a pull-only / whole-disk monitoring gap. Keywords self-monitoring, resource alarm, footprint sampler, growth rate, WAL bytes, page_count, log governor, dedup, durable audit, per-tenant metering. NOT for whole-host infra monitoring (Prometheus/Grafana/node_exporter), APM request tracing, external uptime probes, or one-shot disk-usage reporting.
allowed-tools	Read,Write,Edit,Grep,Glob,Bash(python3:*,node:*,npm:*)
argument-hint	[service-dir] [action: design|implement|audit]
metadata	{"category":"Reliability & Observability","tags":["self-monitoring","resource-alarms","growth-rate","log-governor","failure-visibility"]}

Self-Monitoring Resource Alarms

Make a service watch its OWN resource footprint and alarm before it becomes an incident — instead of discovering the runaway when the disk is already full.

When to Use

✅ Use for:

• A long-running daemon/service that owns a data store (SQLite, embedded KV, append-only log, a table) that could grow without bound.
• Adding a background footprint sampler: own DB bytes, WAL bytes, per-table row counts, and growth rate between samples.
• Wiring graduated warn/crit alarms through a dedup governor so a sustained breach reports once per window, not once per sample.
• Splitting alarms into an ephemeral log signal and a durable audit record.
• Installing global unhandledRejection / uncaughtException (or language equivalent) handlers so a crash is never silent.
• Per-tenant footprint metering in a multi-tenant service; or protecting a dev laptop from a dev-build daemon that bloats its own store.
• Auditing an existing service for the pull-only / whole-disk monitoring gap.

❌ NOT for:

• Whole-host / cluster infra monitoring — use Prometheus, Grafana, node_exporter.
• Request-level APM / distributed tracing — use OpenTelemetry.
• External uptime / synthetic probes.
• One-shot "how full is the disk" reporting with no alarm loop.

---

The Core Insight

Two failures put a real daemon at 313 GB written with zero alarm:

1. Pull-only view — the resource status was computed only when a human opened a panel. Between opens, nobody was watching.
2. Wrong subject — it measured whole-disk percent free, not the daemon's own DB/WAL/table growth. A store that quintupled never tripped anything until the entire volume was nearly full.

The fix is a push sampler that measures your own footprint on a timer, alarms on level and rate, and routes every alarm through a governor so the alarm can never become the spam it exists to catch.

---

Core Process

flowchart TD
  T[Timer fires every ~30s] --> R[Read OWN footprint]
  R --> R1["dbBytes = page_count * page_size"]
  R --> R2["walBytes = stat of -wal file"]
  R --> R3["rows = COUNT per watched table"]
  R1 --> G[Compute growth rate vs previous sample]
  R2 --> G
  R3 --> G
  G --> E{value >= crit?}
  E -->|Yes| C[crit alarm]
  E -->|No| W{value >= warn?}
  W -->|Yes| WA[warn alarm]
  W -->|No| OK[within budget: record sample, done]
  C --> LOG[Route through LogGovernor: dedup + rollup]
  WA --> LOG
  LOG --> DUR{severity == crit?}
  DUR -->|Yes| AUD[ALSO write durable audit record]
  DUR -->|No| END[ephemeral log only]

Two sinks, on purpose — the log is ephemeral, the audit survives rotation:

flowchart LR
  A[Alarm] --> GOV[LogGovernor]
  GOV -->|warn or crit, deduped| L["Ephemeral log, stdout, rotates away"]
  A -->|crit only| D["Durable audit, append-only table, forensics"]

Step 1 — Measure your OWN footprint, never whole-disk

Read what you wrote, behind an injected MetricSources interface so it is testable with no filesystem:

• dbBytes() → page_count * page_size (SQLite pragma), or your store's own size API. Not statvfs / disk-free.
• walBytes() → size of the write-ahead log (0 if checkpointed away).
• rowCount(table) → row count for each watched table.

Wrap each read in a safe() fallback so one failing source cannot abort the whole sample.

Step 2 — Compute growth rate between samples

Keep the previous sample. ratePerSec = (value - prevValue) / dtSec. A store still under its ceiling but climbing megabytes/second is a runaway — the rate alarms before the level does. Persist the rate with every alarm.

Step 3 — Graduated warn/crit thresholds

Absolute ceilings on your footprint (not percentages of a shared disk): warn = comfortable headroom above steady state, crit = the largest footprint you can tolerate before harm. See references/threshold-tuning.md for baselining.

Step 4 — Route every alarm through a dedup governor

The alarm fires on every sample while the breach persists. Without governing, that is thousands of identical lines — the alarm becomes the disk-eater. Use a LogGovernor: first burst occurrences per window emit, the rest are counted and collapsed into one rollup (…and 4,312 more in 5m). The dedup key must be low-cardinality (resource_threshold_crossed:db_bytes:crit) — never embed a timestamp or id, or the governor cannot collapse anything.

Step 5 — Split ephemeral log vs. durable audit

Log the alarm (governed) so an operator sees it live. Also write crit alarms to a durable, append-only audit sink so the record survives log rotation for post-incident forensics. A crit that only ever hit a rotated stdout capture is un-diagnosable — that is how the first incidents stayed mysterious.

Step 6 — Drive it in the background + install failure-visibility handlers

setInterval(sample, 30_000) (call unref() so it never keeps the process alive), and in the same bootstrap register global uncaughtException / unhandledRejection handlers that log and durably audit. A crash with no durable trace is the same blind spot as a footprint with no sampler.

Full copy-paste code for all six steps: references/reference-implementation.md.

---

The Two Horizons

• Multi-tenant — meter footprint per tenant, keyed by tenant id (a structured field you control, safe to embed in the dedup key). Graduated thresholds become quota tiers (warn = approaching plan limit, crit = hard cap), and the same meter feeds usage-based billing.
• Dev-on-dev — the original 313 GB victim was a dev laptop. Ship the monitor enabled by default in dev builds with tighter ceilings, route alarms where a developer actually sees them, and treat a dev crit as a release blocker.

Details in references/threshold-tuning.md.

---

Anti-Patterns

Anti-Pattern: The Pull-Only Resource View

Novice: "We expose a /status endpoint that computes DB size, WAL size, and row counts. Resource monitoring is done." Expert: A status endpoint computes those numbers only when something calls it. Between calls — which, for an internal daemon, may be never until after the incident — nobody is watching. Monitoring is a push activity: a timer samples on its own and alarms without a human in the loop. A pull-only view is a report, not a monitor. Convert it: keep the endpoint for humans, but add a background sampler that raises alarms whether or not anyone is looking. Timeline: A dev-latest daemon exposed exactly this pull-only view and wrote 313 GB before anyone opened the panel. The alarm that would have caught it (a background sampler) did not exist until after the post-mortem. Detection: scripts/audit_self_monitoring.py flags pull-only — footprint reads with no periodic driver (setInterval/cron/scheduler) near them.

Anti-Pattern: Measuring the Whole Disk Instead of Your Own Footprint

Novice: "We alarm when the disk is over 90% full. That covers storage runaways." Expert: Whole-disk percent is the wrong subject. On a big shared volume, your service can 5x its own store — a genuine runaway — while the disk sits at 40%, so nothing trips until you have co-tenanted yourself off a cliff or the whole volume is nearly full. Alarm on your own bytes: page_count * page_size, WAL bytes, per-table row counts, and their growth rate. Whole-disk is a backstop, not the primary signal. Timeline: The 313 GB write storm grew the daemon's own DB and an unrotated stdout capture. Whole-disk monitoring never fired because the volume had room — the daemon's own footprint was the thing exploding, and that was unmeasured. Detection: audit flags wrong-subject — statvfs/disk-free probes present but no own-footprint read.

Anti-Pattern: The Ungoverned Alarm Becomes the Spam

Novice: "When we cross the threshold we log an error every sample so it's impossible to miss." Expert: A sustained breach with a 30s sampler logs an error every 30s forever — and a threshold for storage growth that itself writes unbounded log lines is a self-inflicted version of the very runaway it watches for. Route alarms through a dedup governor: emit the first burst per window, collapse the rest into a single rollup that still reports the true count. You lose redundant bytes, never the fact that it kept happening. Timeline: The same repo had bosun_heartbeat_write_failed and semantic_resolution_failed incidents — a failing op in a tight loop logging a full error object each time: 7,000+ identical lines, a 255 MB stdout capture, a bloated DB. Patched narrowly twice; the class only closed when governed logging became a first-class primitive. Detection: audit flags ungoverned-loop-logging — .error( inside a loop with no governor in the file.

Anti-Pattern: No Global Failure-Visibility Handlers

Novice: "We try/catch around our handlers, so errors are covered." Expert: A rejected promise with no .catch, or a throw in a timer/microtask, escapes every local try/catch. Without a process-level unhandledRejection / uncaughtException handler it prints to a maybe-unrotated stderr and can take the process down with no durable record — the crash and the storage runaway share one blind spot: no one is watching the thing that actually failed. Register global handlers once at bootstrap that log and durably audit, then decide crash-only-restart vs. best-effort-continue deliberately. Timeline: Node treats an unhandled rejection as a hard crash (default since Node 15, 2020). Services carried from older defaults often never added the handlers, so post-Node-15 upgrades turned silent rejections into silent exits. Detection: audit flags no-global-handlers when no such registration exists.

---

Bundle Contents

Path	What it is
references/reference-implementation.md	Copy-paste LogGovernor + SelfMonitor + wiring (all 6 steps). Consult when implementing.
references/threshold-tuning.md	Baselining warn/crit, growth-rate windows, cadence, the two horizons, ephemeral-vs-durable, testing. Consult when tuning or testing.
scripts/audit_self_monitoring.py	Structured code-symbol scan for the four gaps. Run to audit a service: python3 scripts/audit_self_monitoring.py <dir> (exit 1 on any HIGH gap — CI-friendly).

---

Quick Audit

python3 scripts/audit_self_monitoring.py path/to/service --json

Reports: no-footprint-monitor, pull-only, wrong-subject, ungoverned-loop-logging, no-global-handlers. Fix HIGH gaps first, then implement using references/reference-implementation.md.

---

References

Consult these for deep dives — they are NOT loaded by default:

File	Consult When
references/reference-implementation.md	Writing the sampler, governor, durable sink, and background driver
references/threshold-tuning.md	Choosing thresholds, growth-rate rules, multi-tenant metering, dev-on-dev, testing without a filesystem