| name | troubleshoot-apache-httpd |
| description | Use when diagnosing issues with Apache HTTPD: worker exhaustion, memory exhaustion, listen queue overflow, slow backend cascade, or log disk full / pipe stall. Queries Netdata via MCP for process presence, critical-path reachability, requests per second, bytes served per second, request processing duration, applies the diagnostic tree from the Netdata operator playbook, and recommends remediation. |
| version | 0.1.0 |
| author | Netdata |
| license | Apache-2.0 |
| tags | ["netdata","troubleshoot","mcp","apache-httpd"] |
Troubleshoot Apache HTTPD
When to use this skill
- Worker exhaustion: All workers stuck (in W, R, D, or K states). New connections queue then
drop. The most common "server down" symptom.
- Memory exhaustion: MaxRequestWorkers × per-child RSS > available RAM. System swaps, latency
spikes, OOM kills cascade.
- Listen queue overflow: Workers occupied, backlog fills, SYN packets get RST. Server appears
"up" (port open) but unreachable.
- Slow backend cascade: Proxied requests hold workers waiting for slow/dead backends. Workers
fill up from the backend side, not from traffic overload.
- Log disk full / pipe stall: Workers finish requests but block in Logging state. Server appears
alive but serves nothing.
- Graceful restart pile-up: Old-generation children linger during slow request completion.
Multiple overlapping restarts multiply memory consumption.
- Any time the user reports a Apache HTTPD service behaving outside its expected envelope (elevated
errors, latency, saturation, resource exhaustion, or unexpected restarts).
- An on-call engineer is paging on a Netdata alert tied to a Apache HTTPD instance and wants a
structured triage path.
Key facts
- This skill wraps the Netdata operator playbook for Apache HTTPD. It does not replace the playbook;
it routes a coding agent through MCP queries against the same signals the playbook relies on.
- Apache HTTPD is a process-based (or hybrid process+thread) HTTP server whose behavior is
fundamentally shaped by its Multi-Processing Module (MPM). The MPM determines how incoming
connections map to execution units. Knowing which MPM is active is the single most important
prerequisite for interpreting any signal.
- The playbook decomposes Apache HTTPD health into 9 signal domains: Availability, Throughput,
Latency, Errors, Saturation, Resource Utilization. Each domain maps to one rule file in this
skill.
- Dominant failure archetypes the playbook calls out: Worker exhaustion; Memory exhaustion; Listen
queue overflow; Slow backend cascade; Log disk full / pipe stall.
- Netdata observes the signals listed in the rule files via its native collectors, plus any
OpenTelemetry-shipped metrics that your Apache HTTPD instrumentation adds. Both paths end at the
same MCP query surface.
- Netdata's apache collector emits 20 context(s) under
apache.*. The rule files enumerate which
contexts surface which domain; the Verification section below names the load-bearing ones
explicitly.
Step-by-step
- Confirm the Apache HTTPD service is up. Query Netdata via MCP with
list_nodes and filter by the
host running the target. A missing node means the symptom is at the network or orchestrator
layer, not inside the service.
- Pull the last 15 minutes of signals for the target. Use
query_metrics against the contexts
listed in the domain rule files. Run find_anomalous_metrics in parallel over the same window;
anomalies frame which rule file to read first.
- Check for Worker exhaustion. All workers stuck (in W, R, D, or K states). New connections
queue then drop. The most common "server down" symptom. Inspect the rule file whose signals move
first for this mode.
- Check for Memory exhaustion. MaxRequestWorkers × per-child RSS > available RAM. System swaps,
latency spikes, OOM kills cascade. Inspect the rule file whose signals move first for this mode.
- Check for Listen queue overflow. Workers occupied, backlog fills, SYN packets get RST. Server
appears "up" (port open) but unreachable. Inspect the rule file whose signals move first for this
mode.
- Check for Slow backend cascade. Proxied requests hold workers waiting for slow/dead backends.
Workers fill up from the backend side, not from traffic overload. Inspect the rule file whose
signals move first for this mode.
- Check for Log disk full / pipe stall. Workers finish requests but block in Logging state.
Server appears alive but serves nothing. Inspect the rule file whose signals move first for this
mode.
- Correlate with host-level signals (
system.cpu.utilization, system.memory.usage,
system.disk.io_time). Many service-level failures have a host-resource precursor.
- Apply the remediation hinted at in the matching rule file or the operator playbook. Re-run the
MCP queries from the Verification section to confirm the signals returned to expected ranges. A
fix that does not move the signal back is not a fix.
Handy MCP call templates
# Discover metrics from Apache HTTPD
list_metrics with q="apache"
# Pull a specific context over the last window
query_metrics with context="apache.connections", relative_window=-15m
# Rank anomalies for the service or host
find_anomalous_metrics with node=<host> and context_pattern="apache.*"
# Correlate a known problem context with others
find_correlated_metrics around the incident window
# Show current alert state
list_raised_alerts scoped to the node
Common mistakes
- Treating Apache HTTPD as a generic HTTP or process health check. Apache HTTPD has specific failure
archetypes (see Key facts) that generic checks miss.
- Stopping at the first anomalous metric. Several archetypes produce correlated spikes; use
find_correlated_metrics to widen the search before concluding a root cause.
- Quoting percentile latency without the sample count. Low traffic plus a single slow request moves
p99 by seconds.
- Reading dashboards for a window shorter than the failure's fingerprint. Slow-brew failures (queue
growth, bloat, memory fragmentation) need 30+ minutes of data to see the trend.
- Skipping the host-level correlation. A process-level fix for a noisy-neighbour problem does not
hold.
- Assuming alert thresholds are tuned for your workload. Tune against observed Apache HTTPD traffic
before escalating an alert configuration issue.
Verification
Run these MCP queries against the Netdata instance that sees the Apache HTTPD service. Every context
listed below is a real Netdata chart name; the agent does not need to guess.
1. list_metrics filtered by q="apache" (returns every apache.* context Netdata sees)
2. query_metrics with contexts=[apache.connections, apache.conns_async, apache.scoreboard, apache.uptime, apache.requests, apache.reqpersec] and relative_window=-30m
3. find_anomalous_metrics filtered by node=<host> and context_pattern="apache.*"
Load-bearing contexts for this service:
apache.connections: Connections (connections). Dimensions: connections.apache.conns_async: Active Connections (connections). Dimensions: keepalive, closing, writing.apache.scoreboard: Scoreboard (connections). Dimensions: waiting, starting, reading, sending,
keepalive, dns_lookup.apache.uptime: Uptime (seconds). Dimensions: uptime.apache.requests: Requests (requests/s). Dimensions: requests.apache.reqpersec: Lifetime Average Number Of Requests Per Second (requests/s). Dimensions:
requests.
A clean result means every context is within its expected band and the find_anomalous_metrics list
is empty or contains only already-acknowledged items. If the fix was real, re-running the same
queries 10 minutes after applying it will show a clean result. If it does not, revert and look
deeper.
When the fix does not hold
If signals drift back into the anomalous range within 30 minutes of a remediation, the cause was
deeper than the applied change. Typical misdiagnoses for Apache HTTPD:
- Host-resource pressure masquerading as application bug.
- Dependent service (DB, cache, upstream) causing a secondary symptom in the instrumented service.
- Configuration change that was never reloaded (some subsystems only pick up config on full
restart).
Escalate by widening the query window: 2-6 hours instead of 15 minutes. Slow-moving causes are
invisible at triage window sizes.
References
