heteroscale-autoscaling

name	heteroscale-autoscaling
title	HeteroScale Coordinated Autoscaling for Disaggregated LLM Inference
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2508.19559
keywords	["llm-serving","autoscaling","prefill-decode","resource-allocation","gpu-utilization"]
description	Scale disaggregated LLM inference (prefill-decode) via topology-aware scheduling and metric-driven policies, achieving 26.6% GPU utilization improvement and conserving hundreds of thousands GPU-hours daily

Taming the Chaos: Coordinated Autoscaling for Disaggregated LLM Inference

Core Concept

HeteroScale addresses the autoscaling challenge in Prefill-Decode (P/D) disaggregated inference architectures. Traditional autoscaling treats prefill and decode independently, creating bottlenecks. HeteroScale combines topology-aware scheduling (accounting for network and hardware constraints) with a single metric-driven policy that jointly scales both stages, achieving 26.6 percentage point GPU utilization improvement.

Architecture Overview

• Prefill-Decode Disaggregation: Separate resource pools for prompt processing and token generation
• Topology-Aware Scheduler: Routes requests considering hardware heterogeneity and network latency
• Metric-Driven Joint Scaling: Single signal scales both prefill and decode coherently
• Production Infrastructure: Operates on tens of thousands of GPUs
• SLO Preservation: Maintains latency targets while improving utilization

Implementation Steps

Stage 1: Model Prefill-Decode Infrastructure

Understand P/D disaggregated serving architecture.

# Prefill-Decode disaggregated serving
from dataclasses import dataclass
from typing import List, Dict, Tuple
import time

@dataclass
class Request:
    """LLM serving request"""
    request_id: str
    prompt_tokens: int
    output_length: int
    arrival_time: float
    deadline: float  # Latency SLO
    priority: int = 0


@dataclass
class PrefillWorker:
    """Worker specialized for prompt processing"""
    worker_id: str
    gpu_memory: int  # GB
    available_memory: int
    throughput: float  # prompts/sec


@dataclass
class DecodeWorker:
    """Worker specialized for token generation"""
    worker_id: str
    gpu_memory: int
    available_memory: int
    throughput: float  # tokens/sec
    batch_size_limit: int


class DisaggregatedServingCluster:
    """Manage P/D separated serving cluster"""

    def __init__(
        self,
        num_prefill_gpus: int = ,
        num_decode_gpus:  = 
    ):
        .prefill_workers = ._init_workers(num_prefill_gpus, PrefillWorker)
        .decode_workers = ._init_workers(num_decode_gpus, DecodeWorker)

        .request_queue = []
        .prefill_queue = []
        .decode_queue = []

     ():
        
        workers = []
         i  (num_gpus):
            worker = worker_class(
                worker_id=,
                gpu_memory=,  
                available_memory=,
                throughput=  
            )
             worker_class == DecodeWorker:
                worker.batch_size_limit = 
            workers.append(worker)
         workers

     () -> :
        
        start_time = time.time()

        
        prefill_worker = .select_prefill_worker(request)
          prefill_worker:
             {: , : }

        prefill_latency = .prefill_latency(request.prompt_tokens)

        
        decode_workers = .select_decode_workers(request.output_length)
          decode_workers:
             {: , : }

        decode_latency = .decode_latency(request.output_length, (decode_workers))

        total_latency = prefill_latency + decode_latency

         {
            : ,
            : prefill_worker.worker_id,
            : [w.worker_id  w  decode_workers],
            : total_latency,
            : total_latency <= request.deadline
        }

     () -> PrefillWorker:
        
        eligible = [w  w  .prefill_workers  w.available_memory >= ]
          eligible:
             
         (eligible, key= w: (.prefill_queue))

     () -> [DecodeWorker]:
        
        eligible = [w  w  .decode_workers  w.available_memory >= ]
          eligible:
             []

        
        sorted_workers = (eligible, key= w: (.decode_queue))
         sorted_workers[:(, output_length // )]

     () -> :
        
         num_tokens /   

     () -> :
        
         output_length / ( * num_workers)  

Stage 2: Topology-Aware Scheduler

Route requests considering hardware heterogeneity and network constraints.

# Topology-aware scheduling
import networkx as nx

class ClusterTopology:
    """Model cluster network and hardware topology"""

    def __init__(self):
        # Build network graph
        self.topology = nx.DiGraph()
        self.gpu_network_latency = {}  # (gpu1, gpu2) -> latency_ms

    def add_gpu(self, gpu_id: str, rack: str, zone: str):
        """Add GPU to topology"""
        self.topology.add_node(
            gpu_id,
            rack=rack,
            zone=zone,
            latency_to_network=1.0  # ms
        )

    def estimate_latency(self, gpu1: str, gpu2: str) -> float:
        """Estimate network latency between GPUs"""
        node1 = self.topology.nodes[gpu1]
        node2 = self.topology.nodes[gpu2]

        # Same GPU
        if gpu1 == gpu2:
            return 0.0

        # Same rack: low latency
        if node1["rack"] == node2["rack"]:
            return 0.2  # ms

        # Same zone: medium latency
        elif node1[] == node2[]:
               

        
        :
               


 :
    

     ():
        .topology = topology

     () -> :
        
        
        best_prefill_score = ()
        best_prefill = 

         pf_worker  prefill_workers:
            
            queue_cost = ([r  r  request_queue  r.assigned_prefill == pf_worker.worker_id])
            mem_cost = pf_worker.gpu_memory - pf_worker.available_memory
            score =  * queue_cost +  * mem_cost

             score < best_prefill_score:
                best_prefill_score = score
                best_prefill = pf_worker

        
        decode_candidates = []
         d_worker  decode_workers:
            
            network_cost = .topology.estimate_latency(
                best_prefill.worker_id,
                d_worker.worker_id
            )

            queue_cost = ([r  r  request_queue  d_worker  r.assigned_decode])

            total_cost =  * network_cost +  * queue_cost

            decode_candidates.append((d_worker, total_cost))

        
        decode_candidates.sort(key= x: x[])
        selected_decode = [w  w, _  decode_candidates[:(, request.output_length // )]]

         {
            : best_prefill,
            : selected_decode,
            : (
                .topology.estimate_latency(best_prefill.worker_id, d.worker_id)
                 d  selected_decode
            ) / (selected_decode)
        }

Stage 3: Metric-Driven Autoscaling Policy

Single metric scales both prefill and decode stages coherently.

# Metric-driven autoscaling
import math

class AutoscalingMetric:
    """Unified metric for P/D scaling decisions"""

    def __init__(self):
        self.prefill_queue_depth = 0
        self.decode_queue_depth = 0
        self.prefill_utilization = 0.0
        self.decode_utilization = 0.0

    def compute_scaling_signal(self) -> float:
        """
        Compute single metric to drive scaling.

        Signal = f(queue_depth, utilization, latency_headroom)
        Higher signal = scale up
        """
        # Queue depth contribution (0-1)
        max_queue = 1000
        queue_signal = min(
            (self.prefill_queue_depth + self.decode_queue_depth) / max_queue,
            1.0
        )

        # Utilization contribution (0-1)
        avg_utilization = (self.prefill_utilization + self.decode_utilization) / 2
        utilization_signal = avg_utilization

        # Combined signal
        scaling_signal = 0.4 * queue_signal + 0.6 * utilization_signal

        return scaling_signal


class MetricDrivenAutoscaler:
    """Scale P/D resources based on unified metric"""

    def __init__(self, cluster: DisaggregatedServingCluster):
        .cluster = cluster
        .metric = AutoscalingMetric()
        .target_utilization =   

     ():
        
        .metric.prefill_queue_depth = cluster_state.get(, )
        .metric.decode_queue_depth = cluster_state.get(, )
        .metric.prefill_utilization = cluster_state.get(, )
        .metric.decode_utilization = cluster_state.get(, )

     () -> :
        
        signal = .metric.compute_scaling_signal()

        
        scale_up_threshold = 
        scale_down_threshold = 

        decision = {
            : ,
            : ,
            : ,
            : ,
            : signal
        }

         signal > scale_up_threshold:
            
            decision[] = 
            decision[] =   
            decision[] =    

         signal < scale_down_threshold  .can_scale_down():
            
            decision[] = 
            decision[] = -
            decision[] = -

         decision

     () -> :
        
        
         ((.cluster.prefill_workers) <=  
            (.cluster.decode_workers) <= ):
             

        
         .metric.compute_scaling_signal() < 

     ():
        
         decision[] > :
             _  (decision[]):
                .cluster.prefill_workers.append(
                    ._create_prefill_worker()
                )

         decision[] < :
             _  (-decision[]):
                 .cluster.prefill_workers:
                    .cluster.prefill_workers.pop()

         decision[] > :
             _  (decision[]):
                .cluster.decode_workers.append(
                    ._create_decode_worker()
                )

         decision[] < :
             _  (-decision[]):
                 .cluster.decode_workers:
                    .cluster.decode_workers.pop()

     () -> PrefillWorker:
        worker_id = 
         PrefillWorker(
            worker_id=worker_id,
            gpu_memory=,
            available_memory=,
            throughput=
        )

     () -> DecodeWorker:
        worker_id = 
         DecodeWorker(
            worker_id=worker_id,
            gpu_memory=,
            available_memory=,
            throughput=,
            batch_size_limit=
        )

Stage 4: Production Autoscaling Loop

Implement the complete serving loop with autoscaling.

# Production autoscaling loop
import time

class ProductionServing:
    """Production LLM serving with autoscaling"""

    def __init__(self):
        self.cluster = DisaggregatedServingCluster()
        self.topology = ClusterTopology()
        self.scheduler = TopologyAwareScheduler(self.topology)
        self.autoscaler = MetricDrivenAutoscaler(self.cluster)

        self.metrics_history = {
            "gpu_utilization": [],
            "gpu_hours_saved": [],
            "request_latency": []
        }

    def serving_loop(self, duration_seconds: int = 86400):
        """Main serving loop"""
        start_time = time.time()
        check_interval = 60  # Check autoscaling every minute

        while time.time() - start_time < duration_seconds:
            # Process incoming requests
            incoming_requests = self.get_incoming_requests()
            for request in incoming_requests:
                self.process_request(request)

            # Update metrics
            cluster_state = self.measure_cluster_state()
            self.autoscaler.update_metrics(cluster_state)

            # Make scaling decision
            decision = self.autoscaler.make_scaling_decision()

            # Apply scaling
             decision[]  decision[]:
                .autoscaler.apply_scaling(decision)
                ()

            
            .metrics_history[].append(
                cluster_state[]
            )

            
            time.sleep(check_interval)

        
         .generate_report()

     () -> [Request]:
        
         []  

     ():
        
        schedule = .scheduler.place_prefill_and_decode(
            request,
            .cluster.prefill_workers,
            .cluster.decode_workers
        )

     () -> :
        
        prefill_util = (
            (w.gpu_memory - w.available_memory) / w.gpu_memory
             w  .cluster.prefill_workers
        ) / (.cluster.prefill_workers)  .cluster.prefill_workers  

        decode_util = (
            (w.gpu_memory - w.available_memory) / w.gpu_memory
             w  .cluster.decode_workers
        ) / (.cluster.decode_workers)  .cluster.decode_workers  

         {
            : prefill_util,
            : decode_util,
            : (prefill_util + decode_util) / ,
            : (.cluster.prefill_queue),
            : (.cluster.decode_queue)
        }

     () -> :
        
        avg_util = (.metrics_history[]) / (
            .metrics_history[]
        )  .metrics_history[]  

         {
            : avg_util,
            : ,  
            : ,  
            : 
        }

Practical Guidance

Deployment Configuration

• Prefill-Decode Ratio: Maintain 1:2 ratio (1 prefill per 2 decode GPUs)
• Scaling Cooldown: 5-10 minutes between scaling operations to avoid flapping
• Metric Update Frequency: Every 1-2 minutes
• Queue Depth Threshold: Scale up if queue exceeds 800 requests

Network Topology

• Co-locate prefill and decode workers in same racks when possible
• Use high-bandwidth inter-rack links for P/D communication
• Monitor network latency; add capacity if > 5ms between critical paths

When to Use HeteroScale

• Large-scale LLM serving (1000+ GPUs)
• Variable request patterns (scheduling can adapt)
• Cost-sensitive deployments (maximizes GPU utilization)
• Multi-tenant clusters with diverse workloads

When NOT to Use

• Small clusters (<100 GPUs)
• Monolithic LLM serving (not disaggregated)
• Ultra-low latency requirements (autoscaling adds complexity)

Performance Expectations

• GPU Utilization Improvement: +26.6 percentage points
• GPU-Hours Saved: Hundreds of thousands daily at scale
• Latency Impact: Minimal if topology well-designed
• Scaling Overhead: <2% reduction in throughput during scale operations

Reference

Taming the Chaos: Coordinated Autoscaling for Disaggregated LLM Inference. arXiv:2508.19559

• https://arxiv.org/abs/2508.19559