给定一个时间窗口,综合数据库日志、统计视图、扩展插件、操作系统、存储、网络六大维度的证据,重建负载飙升的时间线,溯源根本原因,评估影响面,并给出可落地的规避建议。输出一份可直接用于故障复盘(RCA)的 Markdown 报告。
- 连接与会话状态:
SELECT state, wait_event_type, wait_event, count(*)
FROM pg_stat_activity GROUP BY 1,2,3 ORDER BY count(*) DESC;
SELECT pid, usename, datname, state, wait_event_type, wait_event,
now() - query_start AS running_for, left(query,120) AS query
FROM pg_stat_activity
WHERE state <> 'idle'
ORDER BY running_for DESC;
若窗口内 state = 'active' 且大量堆积、wait_event_type = 'Lock' 集中,指向锁竞争;wait_event_type = 'IO'(如 DataFileRead)集中指向存储瓶颈;wait_event_type = 'Client' 集中通常是应用侧慢/网络慢导致连接被占用而非数据库本身慢。
- 锁等待链(判断阻塞根源):
SELECT blocked_locks.pid AS blocked_pid,
blocking_locks.pid AS blocking_pid,
blocked_activity.query AS blocked_query,
blocking_activity.query AS blocking_query
FROM pg_catalog.pg_locks blocked_locks
JOIN pg_catalog.pg_locks blocking_locks
ON blocking_locks.locktype = blocked_locks.locktype
AND blocking_locks.database IS NOT DISTINCT FROM blocked_locks.database
AND blocking_locks.relation IS NOT DISTINCT FROM blocked_locks.relation
AND blocking_locks.pid != blocked_locks.pid
JOIN pg_catalog.pg_stat_activity blocked_activity ON blocked_activity.pid = blocked_locks.pid
JOIN pg_catalog.pg_stat_activity blocking_activity ON blocking_activity.pid = blocking_locks.pid
WHERE NOT blocked_locks.granted;
- 数据库级吞吐与缓存命中:
SELECT datname, numbackends, xact_commit, xact_rollback,
blks_read, blks_hit,
round(blks_hit::numeric / nullif(blks_hit+blks_read,0), 4) AS hit_ratio,
tup_returned, tup_fetched, temp_files, temp_bytes,
deadlocks, conflicts
FROM pg_stat_database;
hit_ratio 骤降、temp_files/temp_bytes 骤增、deadlocks 非零都是窗口内异常的强信号(需结合外部监控确认是否为窗口内的增量,而非历史累计)。
- 后台写进程/检查点:
SELECT * FROM pg_stat_bgwriter;
SELECT * FROM pg_stat_checkpointer;
checkpoints_req 相对 checkpoints_timed 占比高,说明是被动触发(WAL 写入过快),提示存在写放大或 max_wal_size 偏小。
- 复制状态(主从/多活场景):
SELECT client_addr, state, sent_lsn, write_lsn, flush_lsn, replay_lsn,
write_lag, flush_lag, replay_lag
FROM pg_stat_replication;
SELECT * FROM pg_stat_wal_receiver;
replay_lag 突增指向从库 IO/CPU 跟不上,或主库产生了大事务/大量 WAL。
- 表膨胀与 vacuum 状态:
SELECT relname, n_dead_tup, n_live_tup, last_autovacuum, last_autoanalyze
FROM pg_stat_user_tables
ORDER BY n_dead_tup DESC LIMIT 20;
SELECT * FROM pg_stat_progress_vacuum;