llm-serving-domain

name	llm-serving-domain
description	Domain knowledge on LLM serving systems, QoE metrics, and energy benchmarking — as practiced by Jiachen (Amber) Liu. Covers the Andes QoE framework, ML.ENERGY benchmark, and user-centric serving philosophy. All claims sourced from published papers; unverifiable items marked [unverified].
version	1

LLM Serving Systems — Jiachen Liu's Research Perspective

"As an optimist and strong advocate of AGI, I am building the AI-Native Research Infrastructure so that everyone can be a scientist." — Jiachen (Amber) Liu, University of Michigan / [unverified current affiliation]

---

0. Researcher Profile

Jiachen (Amber) Liu — PhD in Computer Science from University of Michigan, advised by Prof. Mosharaf Chowdhury [unverified advisor]. Research spans MLSys, LLM systems, AI agents, and AI for Science. Published at top venues including NeurIPS (Spotlight), MLSys, TMLR, and arXiv.

Core thesis: LLM serving systems have historically optimized for server-centric metrics (throughput, latency percentiles) while ignoring the metric that actually matters — the user's end-to-end experience. Her work reframes serving as a Quality-of-Experience problem and extends it to energy-aware deployment.

Key publications:

Paper	Year	Venue	Role
Andes: Defining and Enhancing QoE in LLM-Based Text Streaming Services	2024	arXiv 2404.16283	First author
The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimization	2025	NeurIPS 2025 Spotlight	Co-first author (with Jae-Won Chung)
Efficient Large Language Models: A Survey	2023/2024	TMLR 2024	Co-author
Venn: Resource Management for Collaborative Learning Jobs	2025	MLSys 2025	First author
FedTrans: Efficient Federated Learning via Multi-Model Transformation	2024	MLSys 2024	Co-author

[Liu et al., "Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Services", arXiv:2404.16283, 2024] [Chung*, Liu* et al., "The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimization", NeurIPS 2025 Spotlight] [Liu et al., "Efficient Large Language Models: A Survey", TMLR 2024] [Liu et al., "Venn: Resource Management for Collaborative Learning Jobs", MLSys 2025]

---

1. The QoE Framework for LLM Serving

1.1 The Fundamental Misalignment

Existing LLM serving systems optimize metrics that are not aligned with user experience [Liu et al., Andes, §1]. This is the central insight of Jiachen Liu's serving work.

Server-centric metrics and their blind spots:

Metric	What it captures	What it misses
Token generation throughput (tokens/sec)	Aggregate server output	Individual request timelines; no user model
Time-to-First-Token (TTFT)	Initial delay only	Everything after the first token
Average/P90/P99 Time-Per-Output-Token (TPOT)	Statistical summary	Outlier pauses; cascading delays; user reading speed
Request completion latency	End-to-end time	Partial delivery experience; streaming behavior

The key observation: In text streaming services (chatbots, translation, virtual assistants), tokens are incrementally streamed to the user — analogous to video streaming. The user's experience depends on the entire token delivery timeline, not just the first or last token. "Every token matters!" [Liu et al., Andes, §1]

1.2 Text Streaming as an Analogy to Video Streaming

LLM-based conversational AI services provide live user interaction by incrementally streaming text tokens to the user, either as written text or synthesized speech [Liu et al., Andes, §2.1]. The user's interaction timeline has two phases:

1. Initial waiting phase — from request submission to first token delivery. Analogous to video startup delay. TTFT matters here.
2. Token consumption phase — user reads/listens to tokens as they arrive. Analogous to video playback. Token Delivery Speed (TDS) relative to user's consumption speed matters here.

Critical insight on user consumption speed: Users consume tokens at their reading or listening speed, which varies by demographic (age, language) but is bounded. Measured data from the Andes paper [Liu et al., Andes, Figure 2]:

• Average reading speed: ~4.8 tokens/sec (counting 1 word ≈ 1.3 tokens)
• Average listening speed: ~3.3 tokens/sec
• Speed varies by age group and language

Implication: Delivering tokens faster than the user's consumption speed does NOT improve user experience. This excess capacity is wasted — it could be reallocated to serve other requests.

1.3 Formal QoE Definition

Andes defines QoE by comparing the Actual Consumption Timeline against the Ideal Consumption Timeline for each request [Liu et al., Andes, §3.1]:

Ideal Consumption Timeline (T^Ideal): The user's expected experience — low TTFT, with every subsequent token delivered precisely at the user's consumption speed (reading/listening speed).

Actual Consumption Timeline (T^Actual): What actually happens — tokens arrive at times determined by server scheduling, batching, and load.

Four foundational failure cases [Liu et al., Andes, Figure 5]:

1. Perfect experience (QoE = 1): Every token delivered no later than ideal. Delivering earlier than ideal doesn't improve QoE (user can't read faster).
2. Long initial delay: Head-of-line blocking causes long TTFT. Subsequent tokens delivered at proper speed, but the initial delay cascades.
3. Slow streaming speed: Large batch size slows token generation below user's consumption speed. First token on time, but every subsequent token is delayed.
4. Pause during streaming: Preemption pauses token delivery mid-stream. This is insidious — TTFT and average TPOT may look fine, hiding the long pause.

Formal QoE metric [Liu et al., Andes, Equation 3]:

S_delay = Σ(T_i^Actual - T_i^Ideal)      [cumulative delay across all tokens]
S_whole = Σ(T_n^Actual - T_i^Ideal)      [normalizer: total area under actual timeline]
QoE = 1 - S_delay / S_whole

• Perfect delivery: S_delay = 0 → QoE = 1
• Total failure (no tokens arrive): S_delay = S_whole → QoE = 0
• QoE is bounded in [0, 1] for any request state (queued, running, finished)
• Cascading delays are naturally captured: a delay in early token i inflates the delay of all subsequent tokens

Andes's optimization objective: Maximize aggregate QoE across all requests.

---

2. Andes: QoE-Aware LLM Serving System

2.1 System Overview

Andes is a QoE-aware LLM serving system that co-designs the inference server and the text streaming client [Liu et al., Andes, §3.2]. It was built as a proof-of-concept to demonstrate that QoE-aware scheduling dramatically outperforms existing approaches.

Key design principle: Redistribute computation resources across requests over time at token granularity to improve TTFT without affecting the token consumption timeline of other users [Liu et al., Andes, §2.3].

2.2 Architecture

Server-side components [Liu et al., Andes, §3.2, Figure 6]:

1. Request Tracker — Maintains per-request control state:

   • QoE parameters (TTFT target, user token consumption speed)
   • Prompt and partial response
   • Timestamps of each generated token
   • GPU resource usage

2. Token-Level Preemptive Request Scheduler — The core innovation. Dynamically prioritizes requests based on:

   • Expected QoE gain from scheduling this request
   • Current GPU resource usage
   • Risk of QoE degradation

3. LLM Inference Engine — Standard autoregressive generation with preemption support (pause/resume at token boundaries)

Client-side components:

4. Token Pacer — Buffers excess tokens generated by the server and delivers them to the user precisely at the user's consumption speed [Liu et al., Andes, §5]. This decouples server generation speed from user consumption speed, creating scheduling slack.

2.3 Preemptive Scheduling at Token Granularity

The scheduler operates at token granularity (not request granularity), allowing it to preempt a request after any token is generated and switch to another request [Liu et al., Andes, §4].

Three key challenges [Liu et al., Andes, §1]:

(a) Diverse and unpredictable resource demand:

• Requests arrive dynamically with varying prompt lengths, response lengths, and QoE parameters (e.g., user reading speed)
• No one-size-fits-all scheduling policy works

(b) Interdependent aspects in user experience:

• Serving more requests in parallel maximizes GPU memory utilization and reduces TTFT
• But larger batch sizes slow token generation, potentially failing to meet user's ideal token delivery speed
• This tension changes dynamically as input lengths and QoE parameters vary

(c) Token-level preemption overhead:

• Fine-grained scheduling introduces overhead that may degrade throughput
• Must modulate decision frequency and account for preemption/restart costs

Scheduling approach:

• Continuously monitors attained QoE and resource usage per request
• Dynamically prioritizes requests at risk of QoE degradation
• Accounts for both memory and compute constraints
• Incorporates preemption and restart overhead into scheduling decisions

2.4 The Token Pacer

On the client side, Andes introduces a token pacer that temporarily buffers excess tokens and delivers them at the user's consumption speed [Liu et al., Andes, §5]. This is crucial because:

• Server generates tokens much faster than users consume them (e.g., 11 tokens/sec vs. 4.8 tokens/sec reading speed)
• The gap between generation and consumption speed creates scheduling slack
• Pacing decouples server-side scheduling from client-side experience
• Enables the server to preempt and resume requests without the user noticing

Example [Liu et al., Andes, §2.3]: Under moderate load, the server generates 11 tokens/sec while users consume 4.8 tokens/sec. Each request only needs to be served for 1 second every 2.3 seconds. This means the server can potentially serve 2.3× more requests concurrently.

2.5 Push-Based Streaming

Andes's server implements push-based streaming — immediately transmitting tokens to the client as they are generated [Liu et al., Andes, §1]. This is because:

• The server is resource-constrained
• Generated tokens are delivered exclusively to the submitting user
• The client-side token pacer handles smoothing

2.6 Evaluation Results

Models evaluated: Popular LLMs from 3.8B to 70B parameters, including Dense and Mixture-of-Experts architectures, Multi-Head Attention and Grouped-Query Attention [Liu et al., Andes, §6].

Datasets: Three datasets with varying input and output sequence lengths.

Key results [Liu et al., Andes, Abstract]:

• Up to 4.7× improvement in average QoE given the same GPU resources compared to vLLM and Sarathi-Serve
• Up to 61% GPU savings while maintaining the same high QoE

Real-world trace evaluation [Liu et al., Andes, §2.2]:

• Replayed one-hour slice of BurstGPT trace (real-world LLM serving trace)
• vLLM with FCFS scheduling: average TTFT of 10.4 seconds during surges (likely beyond user patience)
• vLLM average TDS: 11.2 tokens/sec (far exceeds any reading speed)
• Andes fixes both by redistributing compute across requests

2.7 Relationship to Existing Systems

Andes was evaluated against [Liu et al., Andes, §6]:

• vLLM [Kwon et al., 2023] — PagedAttention, continuous batching, FCFS
• Sarathi-Serve [Agrawal et al., 2024] — Chunked-prefill scheduling

Both systems use FCFS or simple priority policies that cause head-of-line blocking during load surges. Andes improves on them by:

1. Replacing FCFS with QoE-aware preemptive scheduling
2. Operating at token granularity instead of request granularity
3. Co-designing server and client (token pacer)

---

3. Energy-Aware LLM Serving: The ML.ENERGY Benchmark

3.1 Motivation: The Energy Bottleneck

As generative AI adoption grows explosively, energy has emerged as a critical bottleneck resource [Chung*, Liu* et al., ML.ENERGY, §1]. Key facts:

• ML inference accounts for 80–90% of total compute demand for AI services [Chung*, Liu* et al., ML.ENERGY, §1]
• GPUs account for 50–70% of total provisioned power in datacenters [Chung*, Liu* et al., ML.ENERGY, §2.1]
• Getting access to energy is becoming "increasingly costly, slow, and sometimes impossible" [Chung*, Liu* et al., ML.ENERGY, §1]
• Left unaddressed, the energy bottleneck will "hinder AI research and development progress" and "lead to energy being squeezed out of existing electricity grids" [Chung*, Liu* et al., ML.ENERGY, §1]

The gap: Despite its importance, energy remains a secondary consideration compared to traditional optimization objectives like time and accuracy. ML.ENERGY was created to bridge this gap.

3.2 The ML.ENERGY Initiative

The ML.ENERGY Benchmark is an open-source benchmark suite for measuring inference energy consumption under realistic service environments, with a corresponding Leaderboard at https://ml.energy/leaderboard [Chung*, Liu* et al., ML.ENERGY, §1].

Scope as of early 2025:

• 40 different generative AI model architectures across 6 different tasks
• Tasks include: LLM chat, LLM coding, VLM visual chat, text-to-image, text-to-video, and image-to-video generation
• Built on the Zeus library for programmatic energy measurement [You et al., 2023]

3.3 Four Design Principles

The benchmark is guided by four core principles [Chung*, Liu* et al., ML.ENERGY, §2]:

Principle 1: Generalizability and Portability

• Focus on software-based GPU energy measurement
• GPUs are more standardized than other hardware components
• GPUs allow accurate software-based energy measurement (via NVML/Zeus)
• Measurements are portable across systems using the same GPU model

Principle 2: Representing Real-World Deployments

• Use production-grade software (e.g., vLLM) and hardware (e.g., NVIDIA H100 GPUs)
• Run with workloads representative of real-world use cases
• Measure steady-state energy consumption (when batch size is saturated) to approximate long-term deployment behavior

Principle 3: Energy Measurement at the Right Granularity

• Report energy at the per-request level (whole response), not per-token or per-batch
• Per-request granularity captures model verbosity differences and task- specific output length distributions
• Per-token is too fine (misses higher-level insights); per-batch is too coarse (limited runtime insight)

Principle 4: Actionable Measurement Results

• Provide automated optimization recommendations, not just measurements
• Construct time–energy Pareto frontier across configurations
• Given user-specified latency constraints, recommend energy-optimal configuration and expected savings

3.4 Energy Accounting Methodology

For Diffusion models (simpler case) [Chung*, Liu* et al., ML.ENERGY, §3.2]:

Energy_request = Energy_batch / B

where B = batch size.

For LLM text generation (more complex due to iteration-level batching) [Chung*, Liu* et al., ML.ENERGY, §3.2]:

1. Submit all requests; identify the steady state (batch size saturated at server's maximum configured batch size)
2. Compute average energy per token during steady state
3. Multiply by average output tokens to get per-request energy:

Energy_request = (Energy_steady / Tokens_steady) × (1/N) Σ Tokens_request,i

Why steady state matters: During ramp-up/ramp-down, the server runs with smaller batch sizes and doesn't exhibit the same energy amortization benefits. Steady-state measurements represent well-utilized long-term deployment.

3.5 Automated Optimization Recommendations

The benchmark constructs the time–energy Pareto frontier across all measured configurations [Chung*, Liu* et al., ML.ENERGY, §3.3]:

1. Sweep configurations (GPU model, parallelism, batch size)
2. Measure time and energy for each configuration
3. Build Pareto frontier (no configuration has both lower energy AND lower time)
4. Given user's latency target, select minimum-energy configuration that satisfies the constraint

Latency targets are task-specific:

• LLM chat: Average TPOT should be at least as fast as user's reading/listening speed (connects back to QoE framework!)
• LLM coding: Total generation time for complete code snippet
• Image generation: Total generation time for complete image
• Video generation: Total generation time for complete video

3.6 Key Findings (Early 2025)

Automated optimization can lead to >40% energy savings without changing what is computed by the model [Chung*, Liu* et al., ML.ENERGY, Abstract].

Batch size is a critical configuration that significantly affects both generation time and energy consumption [Chung*, Liu* et al., ML.ENERGY, §3.2]. Sweeping batch size captures varying utilization levels and reveals the time–energy trade-off.

Energy varies significantly across:

• Different model architectures for the same task
• Different tasks for the same model
• Different hardware/software configurations for the same model+task

3.7 Extensibility

The ML.ENERGY Benchmark is designed to be easily extensible [Chung*, Liu* et al., ML.ENERGY, §3.4]:

• Already-supported model families can be extended via fine-tuned variants
• New model architectures can be added by implementing a standard interface
• Custom task datasets can be plugged in
• Open-source on GitHub: https://github.com/ml-energy/benchmark

---

4. Cross-Cutting Themes

4.1 The User-Centric Paradigm Shift

Jiachen Liu's work represents a fundamental paradigm shift in how we think about LLM serving:

Traditional Serving	User-Centric Serving (Liu's approach)
Optimize tokens/sec throughput	Optimize user-perceived QoE
TTFT + average TPOT as metrics	Full token delivery timeline as metric
FCFS scheduling	QoE-aware preemptive scheduling
Server-centric resource allocation	User consumption speed-aware allocation
Treat all tokens equally	"Every token matters"
Energy as afterthought	Energy as first-class metric (ML.ENERGY)

4.2 The QoE–Energy Nexus

There is a natural connection between QoE and energy:

• Over-provisioning wastes energy: Generating tokens faster than users can consume them wastes GPU cycles and energy. QoE-aware scheduling (Andes) naturally reduces energy waste by serving only at the pace users need.
• Energy-optimal configurations for serving: The ML.ENERGY benchmark shows that for chat applications, the latency target should be set to the user's reading/listening speed — exactly the same insight as Andes's QoE framework.
• Batch size as a joint knob: Larger batch sizes increase throughput but slow per-request generation and increase energy. The optimal batch size balances QoE and energy.

4.3 The Streaming Service Analogy

Both Andes and ML.ENERGY treat LLM serving as a streaming service problem:

• Video streaming: Well-studied QoE metrics (startup delay, rebuffering, resolution adaptation). Mature optimization landscape.
• Text streaming (LLM): Andes brings the same user-experience-first thinking to LLM serving. The token pacer is analogous to video buffering.
• Energy streaming: ML.ENERGY treats energy as a resource that flows through the serving pipeline, measurable at per-request granularity.

4.4 From Federated Learning to LLM Serving

Jiachen Liu's earlier work on Venn [Liu et al., MLSys 2025] addresses resource management for collaborative learning jobs at the edge. The throughline:

• Venn: Multi-resource contention among federated learning jobs → efficient edge resource management
• Andes: Multi-user contention for GPU resources → QoE-aware scheduling
• ML.ENERGY: Energy as a contended resource → energy-aware optimization

All three share the philosophy: understand the actual resource demands and user needs, then optimize holistically rather than for single metrics.

---

5. The Broader LLM Serving Ecosystem

5.1 Key Systems for Context

To understand Andes's contributions, know the landscape:

System	Key Innovation	Year	Reference
vLLM	PagedAttention for KV-cache management; continuous batching	2023	Kwon et al., SOSP 2023
Sarathi-Serve	Chunked-prefill for better throughput	2024	Agrawal et al., OSDI 2024
TensorRT-LLM	NVIDIA's optimized inference engine	2023	NVIDIA
SGLang	RadixAttention for prefix caching	2024	Zheng et al.
DeepSpeed-FastGen	SplitFuse for dynamic split of prompts/generations	2024	Microsoft
Andes	QoE-aware preemptive scheduling at token granularity	2024	Liu et al.
DistServe	Disaggregated prefill and decode	2024	Zhong et al., OSDI 2024

5.2 Key LLM Serving Metrics

Standard metrics used in the field:

• TTFT (Time-to-First-Token): Time from request submission to first token delivery. Andes notes this misses everything after the first token.
• TPOT (Time-Per-Output-Token): Time between consecutive token deliveries. Average/P90/P99 variants exist. Andes notes averages miss outlier pauses.
• TDS (Token Delivery Speed): Tokens delivered per second to the user. Andes distinguishes server generation speed from user consumption speed.
• Throughput: Aggregate tokens/sec across all requests. Server-centric; ignores individual request experience.
• SLO Attainment: Fraction of requests meeting Service Level Objectives. Binary pass/fail; doesn't capture degree of degradation.
• QoE (Andes): Holistic metric capturing the full token delivery timeline per request, normalized to [0, 1].

5.3 Key Techniques in Modern Serving

• Continuous Batching (Iteration-level Batching): Unlike static batching, new requests can join a batch mid-execution. Standard in vLLM, Sarathi-Serve. [Yu et al., 2022]
• PagedAttention: Virtual memory-inspired KV-cache management. Enables memory sharing and reduces fragmentation. [Kwon et al., 2023]
• Chunked Prefill: Splitting prompt processing into chunks to reduce TTFT and improve scheduling flexibility. [Agrawal et al., 2024]
• Speculative Decoding: Using a smaller draft model to predict tokens, verified in parallel by the larger model. Reduces latency. [Leviathan et al., 2023; Chen et al., 2023]
• Disaggregated Prefill-Decode: Separating prefill (prompt processing) and decode (token generation) onto different hardware. [Zhong et al., 2024]
• Prefix Caching: Caching KV-cache for common prefixes across requests. [Zheng et al., 2024]

---

6. How to Discuss LLM Serving as Jiachen Liu

6.1 Core Talking Points

When discussing LLM serving from Jiachen's perspective, lead with these ideas:

1. "The metric is wrong." Most serving systems optimize for throughput or simple latency percentiles. These don't capture what users actually experience. QoE is the right metric.

2. "Every token matters." User experience is determined by the entire token delivery timeline, not just the first or last token. A long pause in the middle of streaming degrades QoE even if TTFT and average TPOT look fine.

3. "Users can't read faster than they read." Delivering tokens faster than the user's reading speed doesn't improve experience. This excess capacity is wasted — it's a resource management bug, not a feature.

4. "Energy is not optional." Energy has become a critical bottleneck for AI infrastructure. The ML.ENERGY benchmark makes energy a first-class metric alongside time and accuracy.

5. "User-centric serving is more efficient, not less." Andes achieves up to 4.7× better QoE OR 61% GPU savings. Being user-centric is not at odds with efficiency — it enables it.

6.2 Framing Conversations

When someone mentions throughput optimization: "That's important, but it's a server-centric view. The question is whether that throughput translates to good user experience. A system that generates 100K tokens/sec but leaves users waiting 10 seconds for their first token has a QoE problem."

When someone asks about TTFT/TPOT: "These are necessary but insufficient. TTFT only captures the first moment. Average TPOT can hide long pauses. What we need is a metric that captures the entire token delivery timeline — that's what QoE does."

When energy comes up: "Energy is the next bottleneck. GPUs already account for 50-70% of datacenter power. Our ML.ENERGY benchmark shows that automated optimization can save over 40% energy without changing the model or the output. The key insight: for chat applications, you only need to generate tokens as fast as users can read them — anything faster wastes energy."

When discussing the future of LLM serving: "The field is moving from server-centric to user-centric optimization. The next frontier is jointly optimizing QoE, energy, and cost — treating the user's experience as the primary objective while minimizing resource consumption."

6.3 Technical Depth Signals

To demonstrate deep domain knowledge, reference:

• The cascading delay property of Andes's QoE formula — delays in earlier tokens inflate the delay of all subsequent tokens, naturally captured by S_delay
• The token pacer as a client-side mechanism that creates scheduling slack by buffering server-side over-generation
• The steady-state energy accounting methodology of ML.ENERGY — why ramp-up/ramp-down periods must be excluded from measurements
• The time–energy Pareto frontier construction for automated optimization recommendations
• The connection between user consumption speed (QoE) and energy-optimal batch size (ML.ENERGY) — both set the latency target at reading speed

6.4 Common Questions and Answers

Q: How does QoE relate to SLOs? A: SLO attainment is binary (met/not met). QoE is continuous in [0, 1] — it captures how well the experience was, not just whether a threshold was met. A request with QoE=0.95 is meaningfully better than one with QoE=0.6, even if both technically meet an SLO.

Q: Why preempt at token granularity instead of request granularity? A: Request-granularity preemption wastes the tokens already generated. Token-granularity preemption allows pausing a request mid-generation and switching to another, redistributing GPU cycles where they improve QoE most. The overhead is manageable with careful scheduling.

Q: How does Andes handle unpredictable response lengths? A: The scheduler continuously monitors each request's progress and QoE state. It doesn't need to predict response length — it dynamically reprioritizes based on current QoE risk. Requests approaching QoE degradation get higher priority.

Q: Is the QoE formula specific to text? A: The formula is general to any streaming delivery with a known consumption speed. It could apply to synthesized speech, video frames, or any incremental delivery. The paper focuses on text streaming where reading/listening speed provides the consumption speed parameter.

---

7. Key Papers — Full Bibliography

1. [Liu et al., 2024] Jiachen Liu, Jae-Won Chung, Zhiyu Wu, Fan Lai, Myungjin Lee, Mosharaf Chowdhury. "Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Services." arXiv preprint arXiv:2404.16283, 2024. https://arxiv.org/abs/2404.16283

2. [Chung, Liu et al., 2025]** Jae-Won Chung*, Jeff J. Ma*, Ruofan Wu*, Jiachen Liu*, Oh Jun Kweon, Yuxuan Xia, Zhiyu Wu, Mosharaf Chowdhury. "The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimization." NeurIPS 2025 (Spotlight). arXiv:2505.06371. https://arxiv.org/abs/2505.06371

3. [Liu et al., 2024b] Xinjian Luo, Jiachen Liu, et al. "Efficient Large Language Models: A Survey." Transactions on Machine Learning Research (TMLR), 2024. arXiv:2312.03863. https://arxiv.org/abs/2312.03863

4. [Liu et al., 2025] Jiachen Liu, Ding Ding, Fan Lai, Yiwen Zhang, Mosharaf Chowdhury. "Venn: Resource Management for Collaborative Learning Jobs." MLSys 2025.

5. [Kwon et al., 2023] Woosuk Kwon, Zhuohan Li, Sicheng Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica. "Efficient Memory Management for Large Language Model Serving with PagedAttention." SOSP 2023.

6. [Agrawal et al., 2024] Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Alexey Tumanov, Ramachandran Ramjee. "Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve." OSDI 2024.

7. [Zhong et al., 2024] Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, Hao Zhang. "DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving." OSDI 2024.

8. [Yu et al., 2022] Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, Byung-Gon Chun. "Orca: A Distributed Serving System for Transformer-Based Generative Models." OSDI 2022.

9. [You et al., 2023] Jie You, Jae-Won Chung, Mosharaf Chowdhury. "Zeus: Understanding and Optimizing GPU Energy Consumption of DNN Training." NSDI 2023.

---

8. Glossary

Term	Definition
QoE	Quality of Experience — holistic metric for user-perceived quality of LLM text streaming, defined as 1 - S_delay/S_whole
TTFT	Time-to-First-Token — latency from request submission to first token delivery
TPOT	Time-Per-Output-Token — latency between consecutive token deliveries
TDS	Token Delivery Speed — tokens delivered per second to the user
S_delay	Cumulative delay across all tokens: Σ(T_i^Actual - T_i^Ideal)
S_whole	Normalizer for QoE: Σ(T_n^Actual - T_i^Ideal)
Ideal Consumption Timeline	The user's expected experience — tokens arrive at reading/listening speed
Actual Consumption Timeline	What actually happens — tokens arrive based on server scheduling
Token Pacer	Client-side component that buffers excess tokens and delivers at user's consumption speed
Preemptive Scheduling	Scheduling that can pause mid-request to switch to another request at token granularity
Steady State	Period when batch size is saturated — representative of long-term deployment energy
Pareto Frontier	Set of configurations where no other configuration has both lower energy and lower time
Continuous Batching	Batching technique where new requests can join a batch mid-execution (aka iteration-level batching)
PagedAttention	Virtual memory-inspired KV-cache management technique from vLLM
Zeus	Library for programmatic GPU energy measurement, foundation of ML.ENERGY
Energy-per-Token	Average GPU energy consumed per generated token during steady state
BurstGPT	Real-world LLM serving request trace used in Andes evaluation

---

9. Project Resources

• Andes project page: https://llm-qoe.github.io/
• ML.ENERGY Leaderboard: https://ml.energy/leaderboard
• ML.ENERGY Benchmark (GitHub): https://github.com/ml-energy/benchmark
• Zeus energy measurement library: https://github.com/ml-energy/zeus
• Jiachen Liu's homepage: https://amberljc.github.io/