dualpath-storage-bandwidth-optimization

name	dualpath-storage-bandwidth-optimization
title	DualPath: Breaking Storage Bandwidth Bottleneck in Agentic LLM Inference
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2602.21548
keywords	["LLM inference","distributed serving","KV-cache optimization","bandwidth optimization","multi-turn workloads"]
description	Optimize disaggregated prefill-decoding LLM serving for multi-turn (agentic) workloads by introducing dual-path KV-cache loading. Traditional approach loads all KV-cache to prefill engines, saturating their storage network. DualPath loads to decoding engines first, then transfers via compute network (lower contention). Adaptive routing selects path based on real-time queue depths. Achieves 1.87× offline throughput and 1.96× online serving improvement.

DualPath: Dual-Path KV-Cache Loading for Balanced Storage Network

Disaggregated LLM serving systems separate prefill and decoding operations onto different GPU engines for computational efficiency. However, this architecture creates an asymmetric bottleneck: massive KV-cache tensors must be loaded from storage to prefill engines before processing, saturating the prefill-side storage network. Meanwhile, decoding engines sit underutilized, with available but unused storage bandwidth.

Multi-turn agentic workloads exacerbate this problem: each turn requires loading new KV-caches (contexts from previous turns), creating repeated bandwidth spikes. Standard load balancing treats all storage requests identically, missing the opportunity to redistribute I/O across available resources.

Core Concept

DualPath introduces two loading paths for KV-cache:

Traditional Path: Storage → Prefill Engines (direct, saturates storage NIC)

Novel Path: Storage → Decoding Engines → Prefill Engines (via compute network, distributes load)

The key insight is that compute network bandwidth (RDMA between engines) is abundant and usually underutilized, while storage network bandwidth is the bottleneck. By loading KV-cache to decoding engines first (lower contention), then transferring via compute network, the system distributes I/O load across more resources.

An adaptive controller selects which path to use based on real-time queue lengths and GPU utilization.

Architecture Overview

• Storage Network Monitor: Track utilization of storage-to-prefill links; detect saturation
• Compute Network Monitor: Track compute network (RDMA) utilization; identify available capacity
• Dual-Path Loader: Support both loading paths with switch logic
• Queue Depth Tracker: Monitor storage queue lengths on each engine
• Adaptive Selector: Choose path based on current network state
• QoS Controller: Ensure latency SLAs despite load shifting
• Cost Estimator: Estimate completion time for each path; pick faster one

Implementation

Implement storage path selection logic:

class DualPathKVCacheLoader:
    def __init__(self, num_prefill_engines=4, num_decoding_engines=8):
        .num_prefill_engines = num_prefill_engines
        .num_decoding_engines = num_decoding_engines

        
        .prefill_queue_depths = [] * num_prefill_engines
        .decoding_queue_depths = [] * num_decoding_engines

        
        .storage_network_util = 
        .compute_network_util = 

     ():
        
         path == :
            
            bandwidth =  *   
            contention_factor =  +  * .storage_network_util

            load_time = (kv_cache_size_mb / bandwidth) * contention_factor
             load_time

         path == :
            
            
            phase1_bandwidth =  *  * ( - .compute_network_util)
            phase1_time = (kv_cache_size_mb / phase1_bandwidth) *   

            
            phase2_bandwidth =  *   
            phase2_time = kv_cache_size_mb / phase2_bandwidth

            total_time = phase1_time + phase2_time
             total_time

     ():
        
        time_traditional = .estimate_load_time(kv_cache_size_mb, )
        time_dual = .estimate_load_time(kv_cache_size_mb, )

         deadline_ms:
            
             time_traditional > deadline_ms  time_dual <= deadline_ms:
                 
             time_dual > deadline_ms  time_traditional <= deadline_ms:
                 

        
           time_dual < time_traditional  

     ():
        
        
        .prefill_queue_depths[prefill_engine_id] += 

        
        load_time = .estimate_load_time(kv_cache.size_mb, )
        .storage_network_util +=   

        
        

        
        .prefill_queue_depths[prefill_engine_id] -= 
        .storage_network_util -= 

         kv_cache

     ():
        
        
        least_loaded_decoding = (
            (.num_decoding_engines),
            key= x: .decoding_queue_depths[x]
        )

        
        .decoding_queue_depths[least_loaded_decoding] += 
        load_time_1 = .estimate_load_time(kv_cache.size_mb, ) / 

        
        .prefill_queue_depths[prefill_engine_id] += 
        load_time_2 = .estimate_load_time(kv_cache.size_mb, ) / 

        .decoding_queue_depths[least_loaded_decoding] -= 
        .prefill_queue_depths[prefill_engine_id] -= 

         kv_cache