Use when diagnosing issues with Cockroachdb: lsm compaction death spiral, raft liveness failure, clock skew crisis, hot range, or intent accumulation. Queries Netdata via MCP for Cockroachdb health signals, applies the diagnostic tree from the Netdata operator playbook, and recommends remediation.
Installer avec Codex ou Claude Copiez ce prompt, collez-le dans Codex, Claude ou un autre assistant, puis laissez-le vérifier la page du skill et l'installer pour vous.
Une commande directe contourne le prompt de vérification. Examinez la source avant de l'exécuter.
Use when diagnosing issues with Cockroachdb: lsm compaction death spiral, raft liveness failure, clock skew crisis, hot range, or intent accumulation. Queries Netdata via MCP for Cockroachdb health signals, applies the diagnostic tree from the Netdata operator playbook, and recommends remediation.
Raft Liveness Failure: A node becomes slow (GC pause, disk stall, CPU saturation), can't
process Raft heartbeats, loses leadership, the cluster redistributes
leases causing cascading unavailability windows.
Clock Skew Crisis: NTP misconfiguration causes clock drift. Node self-terminates at >80% of
max-offset. If multiple nodes drift (common with shared NTP), quorum is
lost.
Hot Range: A single range receives disproportionate traffic due to sequential key patterns.
One node bottlenecks the entire workload while others idle.
Intent Accumulation: Long-running or abandoned transactions leave write intents that block
other transactions, cascading latency across the cluster.
Memory Pressure / GC Thrashing: Large queries exhaust SQL memory budget. Go GC pauses exceed
Raft heartbeat interval (3s), causing liveness loss and
oscillating availability.
Any time the user reports a Cockroachdb service behaving outside its expected envelope (elevated
errors, latency, saturation, resource exhaustion, or unexpected restarts).
An on-call engineer is paging on a Netdata alert tied to a Cockroachdb instance and wants a
structured triage path.
Key facts
This skill wraps the Netdata operator playbook for Cockroachdb. It does not replace the playbook;
it routes a coding agent through MCP queries against the same signals the playbook relies on.
CockroachDB is a distributed, strongly-consistent SQL database built on a replicated key-value
store. It layers SQL execution on top of a transactional KV engine that uses Raft consensus for
replication and MVCC for concurrency control. To reason about its failures, you must hold several
interacting subsystems in your head simultaneously.
Dominant failure archetypes the playbook calls out: LSM Compaction Death Spiral; Raft Liveness
Failure; Clock Skew Crisis; Hot Range; Intent Accumulation.
Netdata observes the signals listed in the rule files via its native collectors, plus any
OpenTelemetry-shipped metrics that your Cockroachdb instrumentation adds. Both paths end at the
same MCP query surface.
Netdata's cockroachdb collector emits 60 context(s) under cockroachdb.*. The rule files
enumerate which contexts surface which domain; the Verification section below names the
load-bearing ones explicitly.
Step-by-step
Confirm the Cockroachdb service is up. Query Netdata via MCP with list_nodes and filter by the
host running the target. A missing node means the symptom is at the network or orchestrator
layer, not inside the service.
Pull the last 15 minutes of signals for the target. Use query_metrics against the contexts
listed in the domain rule files. Run find_anomalous_metrics in parallel over the same window;
anomalies frame which rule file to read first.
Check for LSM Compaction Death Spiral. Writes outpace compaction. L0 sublevel count climbs.
Read latency goes exponential. Write stalls follow. The node appears "stuck." Inspect the rule
file whose signals move first for this mode.
Check for Raft Liveness Failure. A node becomes slow (GC pause, disk stall, CPU saturation),
can't process Raft heartbeats, loses leadership, the cluster redistributes leases causing
cascading unavailability windows. Inspect the rule file whose signals move first for this mode.
Check for Clock Skew Crisis. NTP misconfiguration causes clock drift. Node self-terminates at
80% of max-offset. If multiple nodes drift (common with shared NTP), quorum is lost. Inspect the
rule file whose signals move first for this mode.
Check for Hot Range. A single range receives disproportionate traffic due to sequential key
patterns. One node bottlenecks the entire workload while others idle. Inspect the rule file whose
signals move first for this mode.
Check for Intent Accumulation. Long-running or abandoned transactions leave write intents
that block other transactions, cascading latency across the cluster. Inspect the rule file whose
signals move first for this mode.
Correlate with host-level signals (system.cpu.utilization, system.memory.usage,
system.disk.io_time). Many service-level failures have a host-resource precursor.
Apply the remediation hinted at in the matching rule file or the operator playbook. Re-run the
MCP queries from the Verification section to confirm the signals returned to expected ranges. A
fix that does not move the signal back is not a fix.
Handy MCP call templates
# Discover metrics from Cockroachdb
list_metrics with q="cockroachdb"
# Pull a specific context over the last window
query_metrics with context="cockroachdb.process_uptime", relative_window=-15m
# Rank anomalies for the service or host
find_anomalous_metrics with node=<host> and context_pattern="cockroachdb.*"
# Correlate a known problem context with others
find_correlated_metrics around the incident window
# Show current alert state
list_raised_alerts scoped to the node
Common mistakes
Treating Cockroachdb as a generic HTTP or process health check. Cockroachdb has specific failure
archetypes (see Key facts) that generic checks miss.
Stopping at the first anomalous metric. Several archetypes produce correlated spikes; use
find_correlated_metrics to widen the search before concluding a root cause.
Quoting percentile latency without the sample count. Low traffic plus a single slow request moves
p99 by seconds.
Reading dashboards for a window shorter than the failure's fingerprint. Slow-brew failures (queue
growth, bloat, memory fragmentation) need 30+ minutes of data to see the trend.
Skipping the host-level correlation. A process-level fix for a noisy-neighbour problem does not
hold.
Assuming alert thresholds are tuned for your workload. Tune against observed Cockroachdb traffic
before escalating an alert configuration issue.
Verification
Run these MCP queries against the Netdata instance that sees the Cockroachdb service. Every context
listed below is a real Netdata chart name; the agent does not need to guess.
1. list_metrics filtered by q="cockroachdb" (returns every cockroachdb.* context Netdata sees)
2. query_metrics with contexts=[cockroachdb.process_uptime, cockroachdb.sql_connections, cockroachdb.sql_started_dml_statements, cockroachdb.sql_executed_dml_statements, cockroachdb.node_liveness_heartbeats, cockroachdb.sql_errors] and relative_window=-30m
3. find_anomalous_metrics filtered by node=<host> and context_pattern="cockroachdb.*"
cockroachdb.sql_errors: SQL Statements and Transaction Errors (errors). Dimensions: statement,
transaction.
A clean result means every context is within its expected band and the find_anomalous_metrics list
is empty or contains only already-acknowledged items. If the fix was real, re-running the same
queries 10 minutes after applying it will show a clean result. If it does not, revert and look
deeper.
When the fix does not hold
If signals drift back into the anomalous range within 30 minutes of a remediation, the cause was
deeper than the applied change. Typical misdiagnoses for Cockroachdb:
Host-resource pressure masquerading as application bug.
Dependent service (DB, cache, upstream) causing a secondary symptom in the instrumented service.
Configuration change that was never reloaded (some subsystems only pick up config on full
restart).
Escalate by widening the query window: 2-6 hours instead of 15 minutes. Slow-moving causes are
invisible at triage window sizes.
