starrocks-debug-skills

Use when the tablet scheduler is unhealthy: replicas are missing, clone tasks are failing or timing out, balance is too slow (disk usage skewed) or too aggressive (impacting imports), BE decommission is stuck, or colocate groups are unstable. Covers SHOW PROC /cluster_balance analysis, fe_tablet_schedules system table, clone slot tuning, and decommission unblocking via recycle-bin and replica count checks.

Use when compaction score is sustained above 100, imports are failing with 'too many versions', or query MERGE time is elevated. Covers three root causes: import rate exceeding cumulative compaction throughput (Cause A), batch DELETE creating delete-predicate version explosion (Cause B), and large tablet where base compaction is not self-triggering (Cause C). Shared-nothing deployments only — for shared-data see shared-data/SKILL.md.

Use as the first stop when a BE or FE host shows high CPU (elevated load average or top near 100%). Attributes CPU to the right workload — query pipeline, import flush/compression, compaction threads, background tasks, FE GC, or non-StarRocks processes — then hands off to the appropriate deep-dive skill (query/import/compaction/node). Does not resolve root causes directly; acts as a triage and attribution layer.

Use when external catalog queries (Hive, Iceberg, S3, HDFS, Paimon) are failing, hanging, or returning stale data. Covers Hive Metastore connectivity errors, Kerberos authentication failures (keytab expiry, GSS errors, clock skew), S3 rate limiting and SSL issues, network saturation from HMS, and metadata cache TTL tuning for data freshness.

Use when FE or BE processes fail to start or stay alive. Covers port conflicts (YARN co-location), missing meta or storage directories, multi-NIC priority_networks misconfiguration, clock drift causing replica delta errors, BDB journal corruption blocking FE replay, HMS blocking FE startup, and SSL certificate expiry causing partial availability. Includes startup log triage for both FE and BE.

Use when optimizing for high-QPS (thousands of queries/sec) workloads or diagnosing throughput plateaus and latency spikes under load. Covers pipeline_dop=1 tuning, connection pooling (HikariCP/Druid), PreparedStatement plan caching, query cache hit-rate analysis, PK model point-query optimization (short-circuit scan, persistent index), and detecting session-level timeout overrides that cause memory volatility.

Use when import jobs are slow, timing out, or failing — Stream Load, Broker Load, Routine Load, INSERT INTO, Flink connector. Covers write-slow triage (async_delta_writer / memtable_flush / segment_replicate_sync thread pools), publish timeout on PK tables, RPC Failed errors, Reached timeout auto-diagnostics (v3.4+), and load profile analysis. Also covers read-slow causes and Kafka partition bottlenecks.

Use when async materialized views are failing to refresh, timing out, becoming inactive, or query rewrite is not triggering. Covers refresh OOM (spill-to-disk), timeout tuning, MV reactivation, staleness tolerance for query rewrite, sync MV hit verification via EXPLAIN, FE restart impact on refresh schedules, and S3 rate limiting during refresh.

Use when a BE or FE node is abnormal: BE out-of-memory or crash, FE deadlock causing all queries to hang, FE Full GC pauses, or FE heap memory growth and leak. Covers mem_tracker curl analysis, large-memory-alloc log search (query_id → SQL), jstack + DeadLockChecker deadlock tracing, GC log analysis, and memory allocation flame graph interpretation.

Use when queries are slow, hanging, or returning wrong results. Covers the full FE→BE query hang flow, profile-based bottleneck analysis (scan skew, MERGE time dominance, join OOM, runtime filter), predicate pushdown failures, and bug localization via session variable exclusion. Start here for any query performance complaint on a StarRocks cluster.

Use when setting up or troubleshooting resource groups, query queues, or big-query circuit breakers — or when a runaway query is starving other workloads. Covers resource group classifier mismatches (queries falling to default_wg), CPU/memory/scan limit calibration from audit log percentiles, SQL blacklist patterns, query queue full rejection errors, and emergency statistics collection disable when BRPC is saturated.

Use for shared-data (lake-mode / cloud-native) deployments only — not for shared-nothing. Covers DataCache corruption and autoscaling-induced compaction regressions, S3 multipart upload failures and 503 rate limiting, FE leader switch causing in-flight transaction loss, compaction score management via be_cloud_native_compactions, and cached tablet metadata inconsistencies causing publish failures.

Use when investigating tablet-level structural problems: data skew (one tablet 10× larger than peers), too many tablet versions blocking writes (too many versions error), error-state replicas, over-sharding causing high FE metadata overhead, or disk balance retry loops. Covers SHOW TABLET analysis, be_tablets SQL queries, bucket key optimization, and tablet health thresholds (VersionCount, DataSize variance, num_rowset).