clickhouse-incident-runbook

name	clickhouse-incident-runbook
description	ClickHouse incident response — triage, diagnose, and remediate server issues using system tables, kill stuck queries, and execute recovery procedures. Use when ClickHouse is slow, unresponsive, OOM-killed, out of disk, backing up merges, or producing errors in production and you need an on-call playbook. Trigger with "clickhouse incident", "clickhouse outage", "clickhouse down", "clickhouse emergency", "clickhouse on-call", "clickhouse broken".
allowed-tools	Read, Bash(kubectl:*), Bash(curl:*)
version	1.7.0
license	MIT
author	Jeremy Longshore <[e-mail]>
tags	["saas","database","analytics","clickhouse","olap"]
compatibility	Designed for Claude Code

ClickHouse Incident Runbook

Overview

Step-by-step procedures for triaging and resolving ClickHouse incidents using built-in system tables and SQL commands. Start here: assess severity, run quick triage, walk the decision tree, then jump to the matching remediation procedure.

Prerequisites

• Network access to the ClickHouse HTTP interface (default port 8123) or a working clickhouse-client.
• A user with rights to read system.* tables and issue KILL QUERY / ALTER.
• Shell access to the host or container for P1 restarts (systemctl, docker, or kubectl).

Severity Levels

Level	Definition	Response	Examples
P1	ClickHouse unreachable / all queries failing	< 15 min	Server down, OOM, disk full
P2	Degraded performance / partial failures	< 1 hour	Slow queries, merge backlog
P3	Minor impact / non-critical errors	< 4 hours	Single table issue, warnings
P4	No user impact	Next business day	Monitoring gaps, optimization

Instructions

Work the incident top to bottom: triage, classify with the decision tree, then apply the matching procedure.

1. Quick triage (run first)

# 1. Is ClickHouse alive? (8123 is the default ClickHouse HTTP interface port)
curl -sf 'http://localhost:8123/ping' && echo "UP" || echo "DOWN"

# 2. Can it answer a query?
curl -sf 'http://localhost:8123/?query=SELECT+1' && echo "OK" || echo "QUERY FAILED"

# 3. Check ClickHouse Cloud status
curl -sf 'https://status.clickhouse.cloud' |  -5