linux-observability

name	linux-observability
description	Use when adding or diagnosing Prometheus/node_exporter metrics, central log shipping, or lightweight health endpoints; use linux-system-monitoring for an ad hoc host snapshot and linux-log-management for local log analysis.
license	MIT
metadata	{"portable":true,"compatible_with":["claude-code","codex"],"author":"Peter Bamuhigire","author_url":"techguypeter.com","author_contact":"[phone]"}

Linux Observability

Distro support

Prometheus, node_exporter, and Grafana are portable (static binaries / cross-platform packages); the systemd units are identical. The differences are install source and opening the scrape port on the firewall. Body uses Debian/Ubuntu; the RHEL family (Fedora, RHEL, CentOS Stream, Rocky, Alma, Oracle) equivalents are in the matrix.

Concept	Debian/Ubuntu	RHEL family
Install node_exporter	apt / release binary	dnf (EPEL has golang-github-prometheus-node-exporter) / release binary
Open scrape port (9100)	ufw allow 9100/tcp	firewall-cmd --add-port=9100/tcp --permanent && firewall-cmd --reload
Install Telegraf	InfluxData apt repo / apt install telegraf	InfluxData yum repo / dnf install telegraf
Install Datadog agent	official script / Datadog apt repo	official script / Datadog dnf repo
Open Telegraf scrape port (9273)	ufw allow 9273/tcp (restrict to monitor IP)	firewall-cmd --add-rich-rule ... port=9273
systemd unit / service	identical	identical
Log forwarding	rsyslog / journald	same (journald primary on RHEL)
Health endpoint pattern	identical	identical

RHEL-family note: node_exporter is in EPEL on RHEL/Rocky/Alma (ensure_epel), in main on Fedora, or install the upstream static binary on either. Open the firewall with firewalld, not ufw.

In sk-* scripts use the common.sh primitives (pkg_install, ensure_epel, firewall_allow) instead of hardcoding the family. See linux-bash-scripting and docs/multi-distro/plan.md.

Use when

• Adding metrics collection, log shipping, or health endpoints to a server.
• Making a host visible to Prometheus or a central logging pipeline.
• Standardizing /health behavior for operational checks.

Do not use when

• The task is reading local logs only; use linux-log-management.
• The task is general performance triage without telemetry design work; use linux-system-monitoring.

Required Inputs

Artefact	Source	Required?	If absent
Telemetry objective and service	Service owner/SLO	yes	Stop before installing an unfocused collector
Collector or log destination contract	Platform owner	yes	Produce design only; do not expose a port or ship logs
Network, authentication, labels, retention	Platform/security policy	for integration	Mark end-to-end delivery unassessed
Change authority and rollback window	Operations owner	for mutation	Limit to read/search inspection and proposed config

Capability Contract

Read/search access supports current-state assessment. Installing exporters/agents, editing units/config, opening ports, transmitting logs, or reloading services requires explicit authority. Use least-privilege credentials and never expose sensitive health data.

Degraded Mode

Fallback when receiver access is unavailable: validate only the reachable segment and label end-to-end scrape/delivery unassessed. Apply the same limit when network, credentials, root, or binaries are missing; a local listener is not proof of monitoring.

Decision Rules

Choice	Action	Failure avoided
Prometheus can pull securely	Use scrape/exporter model	Unnecessary push state
Logs leave the host	Define filtering, TLS, backpressure, and retention	Data leak or disk exhaustion
Health dependency is slow/optional	Keep liveness cheap; separate readiness	Restart storms
Scrape port is required	Restrict source to collector	Public telemetry exposure

Workflow

1. Define the operational question, consumer, labels, retention, and access boundary.
2. Inspect existing collectors, ports, units, and pipelines before adding components.
3. Choose metrics, logs, or health semantics using the decision table; stop on undefined sensitive fields.
4. Back up config, validate syntax, and apply the smallest authorised change.
5. Verify from the consumer side, including auth, timestamps, labels, and failure state.
6. On failure, recover prior config/service state and report which segment remains unassessed.

Evidence Produced

Artefact	Acceptance
Consumer-side telemetry proof	Shows target, timestamp, expected metric/log/health value, and labels
Configuration evidence	Contains redacted diff, syntax check, service status, and rollback
Coverage statement	Names monitored components, failure states, retention, and unassessed segments

Quality standards

• Observability changes must be testable from the receiving system, not only from the host.
• Keep health endpoints cheap, deterministic, and safe to expose.
• Prefer standard, maintainable telemetry patterns over bespoke one-offs.

Anti-patterns

• Declaring success from a local listener. Fix: verify scrape or receipt at the consumer.
• Shipping logs without destination/retention. Fix: define transport, filtering, backpressure, and expiry.
• Making /health a heavy diagnostic. Fix: keep liveness cheap and separate readiness/dependency detail.
• Opening exporter ports globally. Fix: restrict sources to approved collectors.
• Putting secrets or high-cardinality values in labels. Fix: use bounded non-sensitive dimensions.
• Ignoring pipeline failure behaviour. Fix: test receiver outage, buffering, and recovery.

Outputs

Artefact	Consumer	Acceptance condition
Telemetry integration	Operations	Consumer receives the intended signal with correct labels/timestamps
Operational runbook note	On-call	Names failure signal, retention, access, and recovery steps
Exposure record	Security owner	Documents listener, allowed sources, auth/TLS, and residual risk

Worked Example

For node_exporter on Rocky Linux, restrict TCP 9100 to the Prometheus source, validate the unit, then prove the target is UP and node_uname_info carries the expected instance label. A successful local curl alone leaves consumer delivery unassessed.

References

• ../../docs/continuous-improvement/value-stream-5s-qc-story.md

• ../../docs/continuous-improvement/incident-learning-standard.md

• references/prometheus-setup.md

• references/telemetry-agents.md

• references/log-forwarding.md

• references/health-endpoint-pattern.md

This skill is self-contained. Every step below uses standard tools and released binaries; the body shows Debian/Ubuntu, with RHEL-family equivalents in the Distro support matrix above. The sk-* scripts in the Optional fast path section are convenience wrappers — never required.

This skill owns metrics and log shipping — turning a server from a black box you SSH into occasionally into a first-class citizen of a monitoring stack.

Complements linux-system-monitoring (local, interactive: top, htop, iostat) with remote, continuous, aggregated observability (Prometheus scrape targets, Loki log streams, standard /health endpoints).

Informed by Linux System Administration for the 2020s (observability as mandatory, not optional).

---

When to use

• Installing Prometheus node_exporter on a server.
• Creating a standard /health endpoint for a web service.
• Forwarding logs from a server to a central collector (Loki, ELK, CloudWatch).
• Auditing which metrics and logs a server is currently emitting.

When NOT to use

• top, htop, iostat for right-now triage → linux-system-monitoring.
• Grepping local log files → linux-log-management.
• Prometheus/Grafana alerting rules — those live in the monitoring server's config, not on the managed host.

---

Standing rules

1. Every managed server exposes node_exporter on 127.0.0.1:9100. UFW allows scrape only from the monitoring server's IP — never from 0.0.0.0. Exporter runs as its own unprivileged user.
2. Every web service has a /health endpoint that checks at minimum: database reachable, disk not full, required services running. 200 on healthy, 503 on unhealthy, JSON body with per-check status.
3. Logs ship off the server. Local files are for operator grep, not long-term storage.
4. Metrics have cardinality discipline. No per-request unique IDs as labels. Ever. Prometheus explodes on high cardinality.
5. Public vs internal health endpoints are split. Public /health returns minimal info (200/503); internal /health/detail (behind firewall or auth) returns per-check JSON.
6. TLS on the wire for log shipping. Always. No plain-text logs crossing the network — they contain PII and tokens.

---

Quick reference — manual commands

Install node_exporter from GitHub release (canonical method)

# 1. Download and verify (check releases: https://github.com/prometheus/node_exporter)
VER=1.8.2
cd /tmp
curl -LO https://github.com/prometheus/node_exporter/releases/download/v${VER}/node_exporter-${VER}.linux-amd64.tar.gz
curl -LO https://github.com/prometheus/node_exporter/releases/download/v${VER}/sha256sums.txt
sha256sum -c sha256sums.txt --ignore-missing 2>&1 | grep OK

# 2. Install
tar xzf node_exporter-${VER}.linux-amd64.tar.gz
sudo install -m 0755 node_exporter-${VER}.linux-amd64/node_exporter /usr/local/bin/

# 3. Unprivileged user
sudo useradd --no-create-home --shell /usr/sbin/nologin node_exp

# 4. Systemd unit
sudo tee /etc/systemd/system/node_exporter.service >/dev/null <<'EOF'
[Unit]
Description=Prometheus node exporter
After=network-online.target
Wants=network-online.target

[Service]
User=node_exp
Group=node_exp
ExecStart=/usr/local/bin/node_exporter --web.listen-address=127.0.0.1:9100
Restart=on-failure
RestartSec=3
ProtectSystem=strict
ProtectHome=yes
NoNewPrivileges=yes
CapabilityBoundingSet=
PrivateTmp=yes

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now node_exporter
sudo systemctl status node_exporter --no-pager

# 5. UFW — allow only from monitoring host
sudo ufw allow from 10.0.0.5 to any port 9100 proto tcp

# 6. Verify
curl -s localhost:9100/metrics | head -20

Full deep-dive (collectors, textfile collector, other exporters, systemd hardening options, cardinality discipline) — see references/prometheus-setup.md.

Other telemetry agents (Telegraf, Datadog) — alternatives to node_exporter

node_exporter is the recommended default: OSS, pull-based, minimal attack surface, scraped by your own Prometheus. Two alternatives exist when you have a concrete reason to use them:

Agent	Model	Use when
node_exporter (default)	OSS, pull	You run (or want) your own Prometheus/Grafana stack — keeps the smallest surface.
Telegraf (InfluxData)	OSS, push	You already run InfluxDB, or want one plugin-rich agent (systemd/SNMP/MySQL inputs) pushing to InfluxDB or a Prometheus /metrics endpoint. Stays inside your network.
Datadog agent	SaaS, push	Your org pays for Datadog and wants metrics+logs+APM in one managed platform — and accepts that operational data (and logs, which carry PII/secrets) leaves your network to Datadog's cloud.

Telegraf install (both families, signed repo), inputs (cpu/mem/disk/net/systemd) + outputs (InfluxDB / Prometheus), Datadog install via the official script/repo, datadog.yaml API-key config, enabling integrations, and the privacy/egress consideration of a SaaS agent — all in references/telemetry-agents.md.

API keys / tokens (Telegraf InfluxDB token, Datadog API key) are never put in a world-readable config: store them in a 0600/root systemd env file and inject via the unit — see linux-secrets.

Log forwarding with rsyslog (TLS)

sudo apt install rsyslog-gnutls
sudo mkdir -p /etc/rsyslog.d

# Minimal forwarder config (rsyslog.d/90-forward.conf)
sudo tee /etc/rsyslog.d/90-forward.conf >/dev/null <<'EOF'
$DefaultNetstreamDriver gtls
$DefaultNetstreamDriverCAFile /etc/ssl/certs/loghost-ca.crt
$ActionSendStreamDriverMode 1
$ActionSendStreamDriverAuthMode x509/name
$ActionSendStreamDriverPermittedPeer loghost.internal

$ActionQueueType LinkedList
$ActionQueueFileName fwdRule1
$ActionResumeRetryCount -1
$ActionQueueSaveOnShutdown on

*.* @@loghost.internal:6514
EOF

sudo systemctl restart rsyslog

Full setup with fluent-bit, vector, promtail, and journald forwarding — see references/log-forwarding.md.

Minimal /health endpoint (Nginx → PHP)

# A /health.php file in your web root:
sudo tee /var/www/html/health.php >/dev/null <<'PHP'
<?php
header('Content-Type: application/json');
$checks = [];
$ok = true;

// DB reachable
try {
    $pdo = new PDO('mysql:host=127.0.0.1;dbname=app', 'health', getenv('HEALTH_PASS'));
    $pdo->query('SELECT 1');
    $checks['db'] = 'ok';
} catch (Throwable $e) {
    $checks['db'] = 'fail';
    $ok = false;
}

// Disk < 90% on / and /var
foreach (['/', '/var'] as $mount) {
    $pct = 100 - (disk_free_space($mount) / disk_total_space($mount) * 100);
    $checks["disk:$mount"] = $pct < 90 ? 'ok' : 'fail';
    if ($pct >= 90) $ok = false;
}

http_response_code($ok ? 200 : 503);
echo json_encode(['status' => $ok ? 'ok' : 'fail', 'checks' => $checks]);
PHP

Full pattern (Nginx-only health via stub_status, Node.js variant, caching, auth for detail endpoint) — see references/health-endpoint-pattern.md.

---

Typical workflows

Workflow: "Make this server visible to Prometheus"

1. Install node_exporter (command sequence above).
2. sudo ufw allow from <monitor-ip> to any port 9100 proto tcp.
3. curl -s localhost:9100/metrics | head -20 — confirm metrics.
4. Add the server as a scrape target on your Prometheus server (outside the scope of this skill — that lives on the monitoring host).

Workflow: "Add /health to an existing vhost"

1. Write the health.php (or Node.js equivalent) above.
2. In the Nginx vhost, add:

   location = /health {
       include fastcgi_params;
       fastcgi_pass unix:/run/php/php8.3-fpm.sock;
       fastcgi_param SCRIPT_FILENAME /var/www/html/health.php;
       access_log off;
   }
   

3. sudo nginx -t && sudo systemctl reload nginx.
4. Test: curl -sI https://example.com/health → 200 or 503.

Workflow: "Forward logs to a central Loki collector"

1. Install fluent-bit (apt install fluent-bit or upstream repo for newer versions).
2. Drop /etc/fluent-bit/fluent-bit.conf from references/log-forwarding.md.
3. sudo systemctl enable --now fluent-bit.
4. Verify arrival at the Loki server.

---

Troubleshooting / gotchas

• node_exporter listening on 0.0.0.0 by accident. The systemd unit above pins 127.0.0.1:9100 — verify with ss -tlnp | grep 9100. If it's bound everywhere, fix the --web.listen-address flag and reload.
• UFW allows the scrape but Prometheus can't reach it. Check that the rule is from <monitoring-ip> not to <monitoring-ip>. Easy direction confusion.
• High-cardinality metric explodes Prometheus. Signs: Prometheus OOMs after adding a new exporter. Usually a label with request IDs or user IDs. Drop the offending metric or relabel it away.
• /health page cached by CDN. Set Cache-Control: no-store in the Nginx location. A cached 200 will hide a real 503 from the load balancer for minutes.
• rsyslog forwarding queues on disk fill /var/spool. Set a max queue size on the $ActionQueue* config, and monitor /var/spool/rsyslog.
• Fluent-bit does not automatically rotate its own logs. Configure its log target carefully — it's a log shipper that can create logs.

---

References

• references/prometheus-setup.md — full node_exporter install, other exporters, cardinality discipline, systemd hardening.
• references/telemetry-agents.md — Telegraf (InfluxData) and Datadog agent as alternatives to node_exporter: install on both families, Telegraf inputs/outputs (InfluxDB/Prometheus), Datadog datadog.yaml/API key/integrations, "when to use which", and the privacy/egress consideration of a SaaS agent.
• references/log-forwarding.md — rsyslog/fluent-bit/vector/promtail with TLS forwarding examples.
• references/health-endpoint-pattern.md — full /health pattern with PHP and Node.js examples, internal detail endpoint, caching strategy.
• Book: Linux System Administration for the 2020s — observability is mandatory, not optional.
• Man pages: rsyslogd(8), systemd.service(5).

---

Optional fast path (when sk-* scripts are installed)

Running sudo install-skills-bin linux-observability installs:

Task	Fast-path script
Install node_exporter as unprivileged systemd service	sudo sk-node-exporter-install --monitor-ip <ip>
Install Telegraf (OSS alternative) with metrics inputs + an output	sudo sk-telegraf-setup --output prometheus
Create/verify a /health endpoint for a vhost	sudo sk-health-endpoint --domain <d> --db mysql
Configure log forwarding over TLS	sudo sk-log-forward-setup --collector <host>:<port> --tls

These are optional wrappers around the manual steps above.

Scripts

This skill installs the following scripts to /usr/local/bin/. To install:

sudo install-skills-bin linux-observability

Script	Source	Core?	Purpose
sk-node-exporter-install	scripts/sk-node-exporter-install.sh	no	Install Prometheus node_exporter as unprivileged systemd service, firewall-restrict, verify scrape.
sk-telegraf-setup	scripts/sk-telegraf-setup.sh	no	Install InfluxData Telegraf (OSS alternative to node_exporter) on both families from the signed repo; write host-metrics inputs (cpu/mem/disk/net/systemd) + one output (Prometheus /metrics or InfluxDB v2); firewall-restrict the scrape port; validate with telegraf --test. Token read from the 0600 env file, never the config.
sk-health-endpoint	scripts/sk-health-endpoint.sh	no	Create and verify /health for a vhost: checks db, disk, required services; 200/503 + JSON.
sk-log-forward-setup	scripts/sk-log-forward-setup.sh	no	Configure rsyslog or fluent-bit to forward journald and webserver logs to a central collector over TLS.