flowprefill-scheduling-preemption

name	flowprefill-scheduling-preemption
title	FlowPrefill: Decoupling Preemption from Prefill Scheduling
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2602.16603
keywords	["LLM serving","scheduling","latency optimization","prefill","decoding","head-of-line blocking"]
description	Improve LLM serving under mixed workloads by decoupling execution granularity from scheduling frequency. Operator-level preemption allows fine-grained interruption at natural boundaries (attention, feed-forward layers) without efficiency loss. Event-driven scheduling triggers decisions only on request arrival/completion. Eliminates head-of-line blocking where long requests starve short time-sensitive ones. Achieves 5.6× higher goodput vs. baselines in production workloads.

FlowPrefill: Fine-Grained Preemption Without Efficiency Loss

LLM serving systems face a fundamental trade-off: batch-oriented processing is efficient but creates long latencies for short requests waiting behind long-running ones (head-of-line blocking). Conversely, frequently pausing requests enables responsive preemption but fragments computation and wastes GPU cycles.

Traditional systems are forced to choose: either long fixed batch chunks (efficient but unresponsive) or short chunks (responsive but inefficient). FlowPrefill decouples these concepts: execute in large operator-granularity chunks for efficiency while scheduling (deciding which request runs next) at fine-grained intervals.

Core Concept

FlowPrefill uses two complementary mechanisms:

Operator-Level Preemption: Pause execution at natural operator boundaries (completion of attention/feed-forward layers) rather than arbitrary fixed points. This allows interruption without wasting partial computations. Most operators align with layer boundaries, enabling efficient context switching.

Event-Driven Scheduling: Rather than checking for preemption at fixed frequencies, schedule only when requests arrive or complete. Between these events, execution continues without overhead, minimizing scheduling latency.

The combination allows responsive preemption (reacting to high-priority arrivals) with efficient computation (long execution runs), solving the efficiency-responsiveness trade-off.

Architecture Overview

• Operator Graph: Represent model as DAG of operators (attention, linear, activation); nodes are individual operations
• Preemption Points: Identify safe boundaries between operators where execution can pause
• Priority Queue: Track waiting requests with priorities (e.g., token deadline, priority tier)
• Event Listener: Monitor request arrivals and completions
• Scheduler: On events, select next request to run; execute until next preemption point
• Context Manager: Save/restore execution state (activations, KV-cache) at preemption points
• QoS Controller: Enforce priority policies and latency targets through scheduling decisions

Implementation

Identify preemption-safe boundaries in the model:

def find_preemption_points(model):
    """
    Identify operator boundaries where execution can safely pause.
    Returns list of (layer_idx, operator_name, is_safe)
    """
    preemption_points = []

     layer_idx, layer  (model.layers):
        
         (layer, ):
            preemption_points.append((layer_idx, , ))

        
         (layer, ):
            preemption_points.append((layer_idx, , ))

    
     preemption_points

 ():
    
    
    safe_operators = [, , , ]

     operator_name  safe_operators