troubleshoot-apache-httpd

# Troubleshoot Apache HTTPD ## When to use this skill - **Worker exhaustion**: All workers stuck (in W, R, D, or K states). New connections queue then drop. The most common "server down" symptom. - **Memory exhaustion**: MaxRequestWorkers × per-child RSS > available RAM. System swaps, latency spikes, OOM kills cascade. - **Listen queue overflow**: Workers occupied, backlog fills, SYN packets get RST. Server appears "up" (port open) but unreachable. - **Slow backend cascade**: Proxied requests hold workers waiting for slow/dead backends. Workers fill up from the backend side, not from traffic overload. - **Log disk full / pipe stall**: Workers finish requests but block in Logging state. Server appears alive but serves nothing. - **Graceful restart pile-up**: Old-generation children linger during slow request completion. Multiple overlapping restarts multiply memory consumption. - Any time the user reports a Apache HTTPD service behaving outside its expected envelope (elevated errors, latency, saturation, resource exhaustion, or unexpected restarts). - An on-call engineer is paging on a Netdata alert tied to a Apache HTTPD instance and wants a structured triage path. ## Key facts - This skill wraps the Netdata operator playbook for Apache HTTPD. It does not replace the playbook; it routes a coding agent through MCP queries against the same signals the playbook relies on. - Apache HTTPD is a process-based (or hybrid process+thread) HTTP server whose behavior is fundamentally shaped by its **Multi-Processing Module (MPM)**. The MPM determines how incoming connections map to execution units. Knowing which MPM is active is the single most important prerequisite for interpreting any signal. - The playbook decomposes Apache HTTPD health into 9 signal domains: Availability, Throughput, Latency, Errors, Saturation, Resource Utilization. Each domain maps to one rule file in this skill. - Dominant failure archetypes the playbook calls out: Worker exhaustion; Memory exhaustion; Listen queue overflow; Slow backend cascade; Log disk full / pipe stall. - Netdata observes the signals listed in the rule files via its native collectors, plus any OpenTelemetry-shipped metrics that your Apache HTTPD instrumentation adds. Both paths end at the same MCP query surface. - Netdata's apache collector emits 20 context(s) under `apache.*`. The rule files enumerate which contexts surface which domain; the Verification section below names the load-bearing ones explicitly. ## Step-by-step 1. Confirm the Apache HTTPD service is up. Query Netdata via MCP with `list_nodes` and filter by the host running the target. A missing node means the symptom is at the network or orchestrator layer, not inside the service. 2. Pull the last 15 minutes of signals for the target. Use `query_metrics` against the contexts listed in the domain rule files. Run `find_anomalous_metrics` in parallel over the same window; anomalies frame which rule file to read first. 3. Check for **Worker exhaustion**. All workers stuck (in W, R, D, or K states). New connections queue then drop. The most common "server down" symptom. Inspect the rule file whose signals move first for this mode. 4. Check for **Memory exhaustion**. MaxRequestWorkers × per-child RSS > available RAM. System swaps, latency spikes, OOM kills cascade. Inspect the rule file whose signals move first for this mode. 5. Check for **Listen queue overflow**. Workers occupied, backlog fills, SYN packets get RST. Server appears "up" (port open) but unreachable. Inspect the rule file whose signals move first for this mode. 6. Check for **Slow backend cascade**. Proxied requests hold workers waiting for slow/dead backends. Workers fill up from the backend side, not from traffic overload. Inspect the rule file whose signals move first for this mode. 7. Check for **Log disk full / pipe stall**. Workers finish requests but block in Logging state. Server appears alive but serves nothing. Inspect the rule file whose signals move first for this mode. 8. Correlate with host-level signals (`system.cpu.utilization`, `system.memory.usage`, `system.disk.io_time`). Many service-level failures have a host-resource precursor. 9. Apply the remediation hinted at in the matching rule file or the operator playbook. Re-run the MCP queries from the Verification section to confirm the signals returned to expected ranges. A fix that does not move the signal back is not a fix. ### Handy MCP call templates ```text # Discover metrics from Apache HTTPD list_metrics with q="apache" # Pull a specific context over the last window query_metrics with context="apache.connections", relative_window=-15m # Rank anomalies for the service or host find_anomalous_metrics with node=<host> and context_pattern="apache.*" # Correlate a known problem context with others find_correlated_metrics around the incident window # Show current alert state list_raised_alerts scoped to the node ``` ## Common mistakes - Treating Apache HTTPD as a generic HTTP or process health check. Apache HTTPD has specific failure archetypes (see Key facts) that generic checks miss. - Stopping at the first anomalous metric. Several archetypes produce correlated spikes; use `find_correlated_metrics` to widen the search before concluding a root cause. - Quoting percentile latency without the sample count. Low traffic plus a single slow request moves p99 by seconds. - Reading dashboards for a window shorter than the failure's fingerprint. Slow-brew failures (queue growth, bloat, memory fragmentation) need 30+ minutes of data to see the trend. - Skipping the host-level correlation. A process-level fix for a noisy-neighbour problem does not hold. - Assuming alert thresholds are tuned for your workload. Tune against observed Apache HTTPD traffic before escalating an alert configuration issue. ## Verification Run these MCP queries against the Netdata instance that sees the Apache HTTPD service. Every context listed below is a real Netdata chart name; the agent does not need to guess. ```text 1. list_metrics filtered by q="apache" (returns every apache.* context Netdata sees) 2. query_metrics with contexts=[apache.connections, apache.conns_async, apache.scoreboard, apache.uptime, apache.requests, apache.reqpersec] and relative_window=-30m 3. find_anomalous_metrics filtered by node=<host> and context_pattern="apache.*" ``` Load-bearing contexts for this service: - `apache.connections`: Connections (connections). Dimensions: connections. - `apache.conns_async`: Active Connections (connections). Dimensions: keepalive, closing, writing. - `apache.scoreboard`: Scoreboard (connections). Dimensions: waiting, starting, reading, sending, keepalive, dns_lookup. - `apache.uptime`: Uptime (seconds). Dimensions: uptime. - `apache.requests`: Requests (requests/s). Dimensions: requests. - `apache.reqpersec`: Lifetime Average Number Of Requests Per Second (requests/s). Dimensions: requests. A clean result means every context is within its expected band and the `find_anomalous_metrics` list is empty or contains only already-acknowledged items. If the fix was real, re-running the same queries 10 minutes after applying it will show a clean result. If it does not, revert and look deeper. ### When the fix does not hold If signals drift back into the anomalous range within 30 minutes of a remediation, the cause was deeper than the applied change. Typical misdiagnoses for Apache HTTPD: - Host-resource pressure masquerading as application bug. - Dependent service (DB, cache, upstream) causing a secondary symptom in the instrumented service. - Configuration change that was never reloaded (some subsystems only pick up config on full restart). Escalate by widening the query window: 2-6 hours instead of 15 minutes. Slow-moving causes are invisible at triage window sizes. ## References - [`rules/availability.md`](./rules/availability.md) - [`rules/throughput.md`](./rules/throughput.md) - [`rules/latency.md`](./rules/latency.md) - [`rules/errors.md`](./rules/errors.md) - [`rules/saturation.md`](./rules/saturation.md) - [`rules/resource-utilization.md`](./rules/resource-utilization.md) - [`rules/proxy-mod-proxy.md`](./rules/proxy-mod-proxy.md) - [`rules/tlsssl.md`](./rules/tlsssl.md) - [`rules/internal-state.md`](./rules/internal-state.md) - Netdata operator playbook: the authoritative source material this skill summarizes. - `skills/netdata-mcp-integration/` for the transport setup. - `skills/netdata-otel-setup/` if additional application signals are needed beyond what Netdata collects natively.