# View all import-related thread pool metrics
curl -s "http://<be_ip>:8060/vars" | grep -E "async_delta_writer|memtable_flush|segment_replicate|segment_flush"# View BRPC thread utilization
curl -s "http://<be_ip>:8060/vars" | grep -E "brpc_worker|bthread|tablet_writer"# Loop all BEsfor be in 10.0.0.1 10.0.0.2 10.0.0.3; doecho"=== $be ==="
curl -s "http://$be:8060/vars" | grep -E "async_delta_writer_pending|async_delta_writer_execute|memtable_flush_io"done
Option 2 — SQL (import state)
-- All running and recent import tasksSELECT ID, LABEL, STATE, TYPE, PROGRESS, SCAN_BYTES, SINK_ROWS,
LOAD_START_TIME, LOAD_FINISH_TIME, ERROR_MSG
FROM information_schema.loads
WHERE STATE NOTIN ('FINISHED', 'CANCELLED')
ORDERBY CREATE_TIME DESC;
-- Historical import throughput by minuteSELECT date_trunc('minute', load_finish_time) AS t,
count(*) AS tpm, sum(SCAN_BYTES) AS scan_bytes, sum(sink_rows) AS sink_rows
FROM _statistics_.loads_history
GROUPBY t ORDERBY t DESC LIMIT 10;
Option 3 — FE transaction log (write vs publish duration)
Start with the FE transaction log (finishTransaction) to split write cost vs publish cost — this tells you which phase to focus on.
If async_delta_writer.pending is high, the pool is too small — increase number_tablet_writer_threads. If execute is high with pending low, the bottleneck is within the task (check wait_flush, wait_replica, pk_preload).
If BRPC used ≈ total and Brpc Processing Requests is near-zero during a timeout, the BRPC layer itself is the bottleneck — not the storage layer.
PK tables have an additional pk_preload sub-phase in async_delta_writer.execute — this alone can cause minutes-long write stalls during clone or decommission operations.
Import Slow Signal Reference
Signal / Error
Points to Cause
Phase
Timeout by txn manager
Cause A or E — write or read slow
Write or Read
[E1008]Reached timeout
Cause A or B — storage write timeout (brpc between Coordinator and Executor)
-- Find slow / stuck importsSELECT ID, LABEL, STATE, TYPE,
TIMESTAMPDIFF(SECOND, LOAD_START_TIME, NOW()) AS elapsed_sec,
PROGRESS, SCAN_BYTES, SINK_ROWS, ERROR_MSG
FROM information_schema.loads
WHERE STATE IN ('LOADING', 'PREPARED', 'COMMITED')
ORDERBY elapsed_sec DESC;
Step 1.2 — Determine which phase is slow
# Split write vs publish duration
grep "<label_or_txn_id>" fe.log | grep "finishTransaction"# "write cost: Xms ... publish total cost: Yms"
Phase timing
Next step
write cost >> expected
Phase 2 (identify write bottleneck)
publish total cost >> expected
Phase 3, Cause D
Both normal but overall slow
Phase 2 (check read phase)
Error: RPC Failed
Phase 3, Cause F
Step 1.3 — Scan thread pool health
# Quick health check on all pools
curl -s "http://<be_ip>:8060/vars" \
| grep -E "(async_delta_writer|memtable_flush|segment_replicate_sync)\.(pending|execute|io)"
If any pending metric is non-zero and sustained → pool is saturated → proceed to Phase 2.
Phase 2 — Identify Slow Phase (Read vs Write vs Publish)
async_delta_writer.pending non-zero and sustained, OR memtable_flush.io ≈ execute
LoadChannel.WaitFlushTime or WaitWriterTime high in profile
Cluster CPU and network are not saturated (ruling out Cause B)
Mechanism: Import pipeline writes through async_delta_writer → memtable_flush → disk. When the writer pool or flush pool is saturated, new tablet write tasks queue. The BRPC callback on the coordinator side waits past its deadline. If the disk (HDD) is the IO bottleneck, adding threads makes it worse — reduce concurrency instead.
# Check pool saturation
curl -s "http://<be_ip>:8060/vars" | grep "async_delta_writer"# async_delta_writer_pending: NNN ← non-zero = pool too small# async_delta_writer_execute: NNN ← high with low pending = within-task bottleneck
curl -s "http://<be_ip>:8060/vars" | grep "memtable_flush"# memtable_flush_io: NNN ← if ≈ execute, disk is the bottleneck
-- v3.2+: increase writer threads (all BEs at once)UPDATE information_schema.be_configs SETvalue=32WHERE name ='number_tablet_writer_threads'; -- default 16-- Increase flush threads per disk (if IO util < 70%)UPDATE information_schema.be_configs SETvalue=4WHERE name ='flush_thread_num_per_store'; -- default 2 per disk-- If HDD and IO > 90%: REDUCE threads to improve per-task throughputUPDATE information_schema.be_configs SETvalue=8WHERE name ='number_tablet_writer_threads';
Brpc Processing Requests counter is near 0 during timeout window (server not processing)
netstat -na | grep 8060 shows many connections in CLOSE_WAIT or TIME_WAIT
Mechanism: BRPC connections between coordinator BE and executor BEs become congested or enter a broken state. New RPC calls queue behind stuck connections. The coordinator sees timeout without the executor ever processing the request. This is different from Cause A where the executor receives the call but processes it slowly.
# Check BRPC thread saturation
curl -s "http://<be_ip>:8060/vars" | grep -E "bthread_count|brpc_worker"# Check interface-level latency
curl -s "http://<be_ip>:8060/vars" | grep -E "tablet_writer_(open|add_chunks|add_segment)"# latency-99 high → server-side slow; latency-avg normal but P99 high → spiky# Check TCP connection health
netstat -na | grep 8060 | awk '{print $6}' | sort | uniq -c
# Many CLOSE_WAIT or TIME_WAIT → stale connections
-- Increase RPC timeout
ADMIN SET FRONTEND CONFIG("brpc_send_plan_fragment_timeout_ms" = "180000");
# tcpdump for network-layer analysis (capture on coordinator BE)
tcpdump -i any -n port 8060 -w /tmp/brpc_$(date +%s).pcap
If BRPC is stuck (not recovering): restart the affected BE. Review brpc_connection_pool_size in FE config.
→ Go to Phase 4 to verify recovery
Cause C — Write Slow: PK Index Rebuild
Confirm all match:
async_delta_writer.pk_preload is high (minutes range)
Table is a Primary Key model table
Cluster has active clone or node decommission tasks in progress
Mechanism: During import, the PK model's async_delta_writer must preload (rebuild) the PK index for each tablet before writing. If the tablet is being cloned or decommissioned, the index is not in the local cache and must be rebuilt from scratch — this can take minutes per tablet and blocks the entire write pipeline for that tablet.
# Confirm pk_preload is the bottleneck
curl -s "http://<be_ip>:8060/vars" | grep "pk_preload"# Check if clone tasks are running
grep "clone tablet" be.INFO | tail -20
grep "migration" fe.log | tail -20
-- Skip PK index preload during import (safe: index is rebuilt on next compaction)UPDATE information_schema.be_configs SETvalue= "true"
WHERE name ='skip_pk_preload';
-- Or per-BE-- curl -XPOST "http://<be_ip>:8040/api/update_config?skip_pk_preload=true"-- Enable persistent index to keep PK index on disk (avoids rebuild)ALTER TABLE<pk_table_name>SET ("enable_persistent_index" = "true");
→ Go to Phase 4 to verify recovery
Cause D — Publish Timeout: Compaction Lag
Confirm all match:
FE finishTransaction log shows publish total cost >> write cost
starrocks_be_publish_version_queue_count (shared-nothing) or lake_publish_tablet_version_queuing_count (shared-data) elevated
Table is a Primary Key model table with high upsert rate
Mechanism: PK table publish uses a synchronous apply path (enable_sync_publish): the transaction cannot commit until all replicas have applied the new rowset — including replaying delete vectors against all accumulated rowsets. When compaction lags behind upserts, accumulated rowsets multiply apply time. Each publish blocks waiting for apply to complete. Subsequent imports queue behind pending publishes.
# Monitor publish queue depth
curl -s "http://<be_ip>:8040/metrics" | grep "publish_version_queue_count"# Check apply latency in BE log
grep "apply_rowset_commit finish" be.INFO | tail -20
# Format: apply_rowset_commit finish. tablet=<id> cost=<ms>ms# Cost > 5000ms per commit → severe backlog
-- Increase publish worker threads-- (be.conf: transaction_publish_version_worker_count, default = #CPU cores)UPDATE information_schema.be_configs SETvalue=16WHERE name ='transaction_publish_version_worker_count';
-- Check PK compaction score (see compaction skill for full diagnosis)-- High update compaction score → run manual compaction-- curl -XPOST "http://<be_ip>:8040/api/compact?compaction_type=update&tablet_id=<tablet_id>"
Also check for clone tasks interfering with publish:
grep "clone.*tablet" be.INFO | tail -20
→ Go to Phase 4 to verify recovery
Cause E — Read Slow: Source Side Bottleneck
Confirm all match:
Profile shows CONNECTOR_SCAN / FileScanNode time >> OLAP_TABLE_SINK time
OR Routine Load task count equals Kafka partition count and lag grows monotonically
Mechanism: The read phase (source → StarRocks) is the bottleneck. For Routine Load, parallelism is capped by Kafka partition count — adding StarRocks resources provides no relief. For Broker Load, many small files cause excessive open/seek overhead. For Stream Load with large JSON batches, deserialization is the CPU bottleneck.
Import Type
Common Causes
Solutions
Stream Load
HTTP client to StarRocks network slow; JSON format with large batches
Reduce batch size; try CSV format
Routine Load
Small batch size; too few Kafka partitions
Increase max_routine_load_batch_size and routine_load_task_consume_second; add Kafka partitions
For Flink connector: check Flink CPU utilization first — it is often the bottleneck, not StarRocks.
sink.buffer-flush.max-bytes
sink.buffer-flush.max-rows
sink.buffer-flush.interval-ms
checkpoint-interval
→ Go to Phase 4 to verify recovery
Cause F — RPC Failed: Statistics Collection Conflict
Confirm all match:
Import ERROR_MSG contains RPC Failed
Failure pattern is intermittent, correlated with scheduled statistics collection windows
FE log shows many concurrent ANALYZE tasks in information_schema.task_runs
Mechanism: Automatic ANALYZE triggers large-scale statistics scans across many tables. These scans saturate the BRPC send queue on BEs. Incoming plan fragment delivery for import tasks fails because the BRPC worker queue is full. FE retries exhaust or time out and the import transaction aborts.
# Check brpc latency during the failure window
curl -s "http://<be_ip>:8060/vars" | grep exec_
# Check TCP connection state on BRPC port
netstat -na | grep 8060 | head -30
-- Immediate mitigation: disable automatic statistics collection
ADMIN SET FRONTEND CONFIG("enable_collect_full_statistic" = "false");
ADMIN SET FRONTEND CONFIG("enable_statistic_collect" = "false");
ADMIN SET FRONTEND CONFIG("enable_statistic_collect_on_first_load" = "false");
-- Increase RPC timeout to tolerate BRPC congestion
ADMIN SET FRONTEND CONFIG("brpc_send_plan_fragment_timeout_ms" = "180000");
-- Long-term: tune statistics collection schedule-- Increase interval and reduce concurrency
ADMIN SET FRONTEND CONFIG("statistic_collect_interval_sec" = "1200");
ADMIN SET FRONTEND CONFIG("statistic_collect_concurrency" = "1");
→ Go to Phase 4 to verify recovery
Phase 4 — Verify Recovery
-- Confirm import tasks are completing successfullySELECT STATE, COUNT(*) AS cnt, AVG(TIMESTAMPDIFF(SECOND, LOAD_START_TIME, LOAD_FINISH_TIME)) AS avg_sec
FROM information_schema.loads
WHERE LOAD_FINISH_TIME > DATE_SUB(NOW(), INTERVAL10MINUTE)
GROUPBY STATE;
-- FINISHED should dominate; CANCELLED/FAILED should be 0-- Confirm no new import failuresSELECT LABEL, STATE, ERROR_MSG
FROM information_schema.loads
WHERE STATE ='CANCELLED'ORDERBY LOAD_FINISH_TIME DESC
LIMIT 10;
-- Confirm publish queue is draining (PK tables)-- Monitor over 2-3 minutes: count should decreaseSELECTCOUNT(*) AS pending_publish
FROM information_schema.loads
WHERE STATE ='COMMITED';
# Confirm thread pool metrics are back to normal
curl -s "http://<be_ip>:8060/vars" | grep "async_delta_writer_pending"# Should be 0 or near-0
curl -s "http://<be_ip>:8060/vars" | grep "memtable_flush_io"# Should be proportional to actual import load (not 100%)# Confirm no new timeout errors in BE log
grep "Reached timeout\|RPC Failed" be.WARNING | tail -10
# Should be empty or timestamps should be in the past
Grafana: starrocks_be_publish_version_queue_count should return to near-0. Import throughput (sink_rows per minute) should return to baseline.
Thread Pool Configuration Reference
Shared-Nothing Write Pipeline
Parameter
Default
Description
Dynamic
number_tablet_writer_threads
16
async_delta_writer pool size
Yes (v2.5+)
flush_thread_num_per_store
2 per disk
memtable_flush / segment_replicate_sync / segment_flush pool size
Yes (v2.5+)
brpc_num_threads
#CPU cores
BRPC worker thread count
No (restart)
write_buffer_size
100MB
Memtable size before flush trigger
Yes (v2.5+)
transaction_publish_version_worker_count
#CPU cores
Publish version worker threads
No (restart)
load_process_max_memory_limit_bytes
100G
Upper bound for import memory
No (restart)
load_process_max_memory_limit_percent
30%
Actual limit = mem_limit * 90% * 30%
No (restart)
skip_pk_preload
false
Skip PK index rebuild during import
Yes (v2.5+)
async_load_task_pool_size
10
Broker Load loading thread pool
No (restart)
Shared-Data Write Pipeline
Parameter
Default
Description
Dynamic
lake_flush_thread_num_per_store
2 × #CPU
memtable flush pool for shared-data
Yes (v2.5+)
Applying Dynamic Parameters
-- v3.2+: apply to all BEs at onceUPDATE information_schema.be_configs SETvalue=<value>WHERE name ='<param>';
SELECT*FROM information_schema.be_configs WHERE name ='<param>'; -- verify
# v2.5+: apply per BE
curl -XPOST "http://<be_ip>:<be_http_port>/api/update_config?<param>=<value>"
curl "http://<be_ip>:<be_http_port>/varz" | grep <param> # verify
Static parameters must be set in be.conf and require restart.
case-007-memory-tracking-leak — slow imports caused by memory tracker leak
case-009-stream-load-stuck — tablet meta cache bug
Causal Chains
Chain 1: async_delta_writer Pool Saturation → Write Timeout
High-concurrency imports saturate async_delta_writer thread pool
↓ observable: BE pool metrics show active ≈ total with queue > 0 for async_delta_writer
New delta write tasks queue; flush cannot proceed
↓ observable: BE log shows wait_flush duration growing per tablet write
BRPC callback exceeds deadline waiting for write acknowledgement
↓ observable: BE log "Reached timeout" on tablet write RPC; import task STATE stuck in LOADING
FE times out waiting for tablet commit
Import job fails with Reached timeout; data not committed.
Trigger conditions: write_buffer_size too small causing excessive flush frequency; import concurrency exceeds thread pool capacity.
Automatic ANALYZE triggers large-scale statistics scan across many tables
↓ observable: FE log shows many concurrent ANALYZE tasks in information_schema.task_runs
BRPC send queue on BE fills; plan fragment delivery fails
↓ observable: BE log "plan fragment send fail"; FE log "RPC Failed" sending fragments to BE
Import fragment cannot reach BE; FE retries exhaust or time out
↓ observable: information_schema.loads shows STATE = FAILED with error containing "RPC Failed"
Import transactions abort
Import jobs fail intermittently during scheduled statistics collection windows.
Break point: Separate statistics schedule from peak import windows; reduce statistic_collect_concurrency; disable via ADMIN SET FRONTEND CONFIG("enable_collect_full_statistic" = "false").
High-frequency upserts on PK table; compaction cannot keep pace
↓ observable: starrocks_be_publish_version_queue_count elevated and growing; num_rowset per tablet rising
Apply thread must replay many delete vectors against accumulated rowsets
↓ observable: BE log shows pk_preload latency increasing; wait_replica extends per commit
Publish version step blocks waiting for apply to complete
↓ observable: FE finishTransaction log shows publish cost >> write cost
Transaction publish deadline exceeded
Import fails with publish timeout; subsequent imports on the same table also stall.
Trigger conditions: PK table with high upsert rate; pk_index_cache_capacity too small; max_pk_compaction_threads not scaled.
Break point: Increase max_pk_compaction_threads; enlarge pk_index_cache_capacity; monitor starrocks_be_publish_version_queue_count as leading indicator.
Long-running BE process accumulates unreleased memory in tracker
↓ observable: SHOW PROC '/current_queries' shows cumulative memUsageBytes not decreasing after query completion
Memory tracker reports usage near mem_limit; import admission control activates
↓ observable: BE log shows import tasks throttled or rejected with "memory exceed"
Import tasks queue but cannot start; progress metric frozen
↓ observable: information_schema.loads shows STATE = LOADING with PROGRESS barely advancing over minutes
Import times out or is manually cancelled
Import appears to start successfully but makes no progress; restarting BE temporarily resolves until leak accumulates again.
Trigger conditions: Memory tracker reference-count bug in a specific code path; BE running for days without restart.
Break point: Audit SHOW PROC '/current_queries' memory; rolling restart affected BE; patch tracker release on error exit paths.
Chain 5: Routine Load Kafka Partition Bottleneck → Lag Grows
Routine Load task count capped by Kafka partition count
↓ observable: SHOW ROUTINE LOAD shows TaskRunning == Kafka partition count; cannot increase parallelism
Incoming message rate exceeds drain rate; lag grows
↓ observable: Kafka consumer group lag metric grows monotonically; PROGRESS in information_schema.loads stalls
Unconsumed messages accumulate in Kafka topic
↓ observable: kafka-consumer-groups.sh --describe shows LAG increasing per partition
Routine Load lag grows unboundedly; data freshness SLA breached
Downstream queries see stale data; eventually Kafka retention boundary reached and messages dropped.
Trigger conditions: Topic created with too few partitions; data volume grew after initial sizing.
Break point: Increase Kafka topic partition count; raise desired_concurrent_number in Routine Load job after repartitioning.
Cross-Skill Guides
guides/cascade-import-rpc-failed.md — Statistics collection → BRPC starvation → import/query RPC failure. Read this when imports fail with "RPC Failed" and ANALYZE tasks are running concurrently.
