| name | clickhouse-incident-runbook |
| description | ClickHouse incident response — triage, diagnose, and remediate server issues
using system tables, kill stuck queries, and execute recovery procedures.
Use when ClickHouse is slow, unresponsive, OOM-killed, out of disk, backing up
merges, or producing errors in production and you need an on-call playbook.
Trigger with "clickhouse incident", "clickhouse outage", "clickhouse down",
"clickhouse emergency", "clickhouse on-call", "clickhouse broken".
|
| allowed-tools | Read, Bash(kubectl:*), Bash(curl:*) |
| version | 1.7.0 |
| license | MIT |
| author | Jeremy Longshore <jeremy@intentsolutions.io> |
| tags | ["saas","database","analytics","clickhouse","olap"] |
| compatibility | Designed for Claude Code |
ClickHouse Incident Runbook
Overview
Step-by-step procedures for triaging and resolving ClickHouse incidents using
built-in system tables and SQL commands. Start here: assess severity, run quick
triage, walk the decision tree, then jump to the matching remediation procedure.
Prerequisites
- Network access to the ClickHouse HTTP interface (default port
8123) or a
working clickhouse-client.
- A user with rights to read
system.* tables and issue KILL QUERY / ALTER.
- Shell access to the host or container for P1 restarts (
systemctl, docker,
or kubectl).
Severity Levels
| Level | Definition | Response | Examples |
|---|
| P1 | ClickHouse unreachable / all queries failing | < 15 min | Server down, OOM, disk full |
| P2 | Degraded performance / partial failures | < 1 hour | Slow queries, merge backlog |
| P3 | Minor impact / non-critical errors | < 4 hours | Single table issue, warnings |
| P4 | No user impact | Next business day | Monitoring gaps, optimization |
Instructions
Work the incident top to bottom: triage, classify with the decision tree, then
apply the matching procedure.
1. Quick triage (run first)
curl -sf 'http://localhost:8123/ping' && echo "UP" || echo "DOWN"
curl -sf 'http://localhost:8123/?query=SELECT+1' && echo "OK" || echo "QUERY FAILED"
curl -sf 'https://status.clickhouse.cloud' | -5
