| name | llm-serving-domain |
| description | Domain knowledge on LLM serving systems, QoE metrics, and energy benchmarking — as practiced by Jiachen (Amber) Liu. Covers the Andes QoE framework, ML.ENERGY benchmark, and user-centric serving philosophy. All claims sourced from published papers; unverifiable items marked [unverified].
|
| version | 1 |
LLM Serving Systems — Jiachen Liu's Research Perspective
"As an optimist and strong advocate of AGI, I am building the AI-Native
Research Infrastructure so that everyone can be a scientist." — Jiachen
(Amber) Liu, University of Michigan / [unverified current affiliation]
0. Researcher Profile
Jiachen (Amber) Liu — PhD in Computer Science from University of Michigan,
advised by Prof. Mosharaf Chowdhury [unverified advisor]. Research spans
MLSys, LLM systems, AI agents, and AI for Science. Published at top venues
including NeurIPS (Spotlight), MLSys, TMLR, and arXiv.
Core thesis: LLM serving systems have historically optimized for
server-centric metrics (throughput, latency percentiles) while ignoring the
metric that actually matters — the user's end-to-end experience. Her work
reframes serving as a Quality-of-Experience problem and extends it to
energy-aware deployment.
Key publications:
| Paper | Year | Venue | Role |
|---|
| Andes: Defining and Enhancing QoE in LLM-Based Text Streaming Services | 2024 | arXiv 2404.16283 | First author |
| The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimization | 2025 | NeurIPS 2025 Spotlight | Co-first author (with Jae-Won Chung) |
| Efficient Large Language Models: A Survey | 2023/2024 | TMLR 2024 | Co-author |
| Venn: Resource Management for Collaborative Learning Jobs | 2025 | MLSys 2025 | First author |
| FedTrans: Efficient Federated Learning via Multi-Model Transformation | 2024 | MLSys 2024 | Co-author |
[Liu et al., "Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Services", arXiv:2404.16283, 2024]
[Chung*, Liu* et al., "The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimization", NeurIPS 2025 Spotlight]
[Liu et al., "Efficient Large Language Models: A Survey", TMLR 2024]
[Liu et al., "Venn: Resource Management for Collaborative Learning Jobs", MLSys 2025]
1. The QoE Framework for LLM Serving
1.1 The Fundamental Misalignment
Existing LLM serving systems optimize metrics that are not aligned with user
experience [Liu et al., Andes, §1]. This is the central insight of Jiachen
Liu's serving work.
Server-centric metrics and their blind spots:
| Metric | What it captures | What it misses |
|---|
| Token generation throughput (tokens/sec) | Aggregate server output | Individual request timelines; no user model |
| Time-to-First-Token (TTFT) | Initial delay only | Everything after the first token |
| Average/P90/P99 Time-Per-Output-Token (TPOT) | Statistical summary | Outlier pauses; cascading delays; user reading speed |
| Request completion latency | End-to-end time | Partial delivery experience; streaming behavior |
The key observation: In text streaming services (chatbots, translation,
virtual assistants), tokens are incrementally streamed to the user — analogous
to video streaming. The user's experience depends on the entire token delivery
timeline, not just the first or last token. "Every token matters!"
[Liu et al., Andes, §1]
1.2 Text Streaming as an Analogy to Video Streaming
LLM-based conversational AI services provide live user interaction by
incrementally streaming text tokens to the user, either as written text or
synthesized speech [Liu et al., Andes, §2.1]. The user's interaction timeline
has two phases:
- Initial waiting phase — from request submission to first token delivery.
Analogous to video startup delay. TTFT matters here.
- Token consumption phase — user reads/listens to tokens as they arrive.
Analogous to video playback. Token Delivery Speed (TDS) relative to user's
consumption speed matters here.
Critical insight on user consumption speed: Users consume tokens at their
reading or listening speed, which varies by demographic (age, language) but is
bounded. Measured data from the Andes paper [Liu et al., Andes, Figure 2]:
- Average reading speed: ~4.8 tokens/sec (counting 1 word ≈ 1.3 tokens)
- Average listening speed: ~3.3 tokens/sec
- Speed varies by age group and language
Implication: Delivering tokens faster than the user's consumption speed
does NOT improve user experience. This excess capacity is wasted — it could be
reallocated to serve other requests.
1.3 Formal QoE Definition
Andes defines QoE by comparing the Actual Consumption Timeline against the
Ideal Consumption Timeline for each request [Liu et al., Andes, §3.1]:
Ideal Consumption Timeline (T^Ideal): The user's expected experience —
low TTFT, with every subsequent token delivered precisely at the user's
consumption speed (reading/listening speed).
Actual Consumption Timeline (T^Actual): What actually happens — tokens
arrive at times determined by server scheduling, batching, and load.
Four foundational failure cases [Liu et al., Andes, Figure 5]:
- Perfect experience (QoE = 1): Every token delivered no later than
ideal. Delivering earlier than ideal doesn't improve QoE (user can't
read faster).
- Long initial delay: Head-of-line blocking causes long TTFT. Subsequent
tokens delivered at proper speed, but the initial delay cascades.
- Slow streaming speed: Large batch size slows token generation below
user's consumption speed. First token on time, but every subsequent token
is delayed.
- Pause during streaming: Preemption pauses token delivery mid-stream.
This is insidious — TTFT and average TPOT may look fine, hiding the
long pause.
Formal QoE metric [Liu et al., Andes, Equation 3]:
S_delay = Σ(T_i^Actual - T_i^Ideal) [cumulative delay across all tokens]
S_whole = Σ(T_n^Actual - T_i^Ideal) [normalizer: total area under actual timeline]
QoE = 1 - S_delay / S_whole
- Perfect delivery: S_delay = 0 → QoE = 1
- Total failure (no tokens arrive): S_delay = S_whole → QoE = 0
- QoE is bounded in [0, 1] for any request state (queued, running, finished)
- Cascading delays are naturally captured: a delay in early token i inflates
the delay of all subsequent tokens
Andes's optimization objective: Maximize aggregate QoE across all requests.
2. Andes: QoE-Aware LLM Serving System
2.1 System Overview
Andes is a QoE-aware LLM serving system that co-designs the inference server
and the text streaming client [Liu et al., Andes, §3.2]. It was built as a
proof-of-concept to demonstrate that QoE-aware scheduling dramatically
outperforms existing approaches.
Key design principle: Redistribute computation resources across requests
over time at token granularity to improve TTFT without affecting the token
consumption timeline of other users [Liu et al., Andes, §2.3].
2.2 Architecture
Server-side components [Liu et al., Andes, §3.2, Figure 6]:
-
Request Tracker — Maintains per-request control state:
- QoE parameters (TTFT target, user token consumption speed)
- Prompt and partial response
- Timestamps of each generated token
- GPU resource usage
-
Token-Level Preemptive Request Scheduler — The core innovation.
Dynamically prioritizes requests based on:
- Expected QoE gain from scheduling this request
- Current GPU resource usage
- Risk of QoE degradation
-
LLM Inference Engine — Standard autoregressive generation with
preemption support (pause/resume at token boundaries)
Client-side components:
- Token Pacer — Buffers excess tokens generated by the server and
delivers them to the user precisely at the user's consumption speed
[Liu et al., Andes, §5]. This decouples server generation speed from
user consumption speed, creating scheduling slack.
2.3 Preemptive Scheduling at Token Granularity
The scheduler operates at token granularity (not request granularity),
allowing it to preempt a request after any token is generated and switch to
another request [Liu et al., Andes, §4].
Three key challenges [Liu et al., Andes, §1]:
(a) Diverse and unpredictable resource demand:
- Requests arrive dynamically with varying prompt lengths, response lengths,
and QoE parameters (e.g., user reading speed)
- No one-size-fits-all scheduling policy works
(b) Interdependent aspects in user experience:
- Serving more requests in parallel maximizes GPU memory utilization and
reduces TTFT
- But larger batch sizes slow token generation, potentially failing to meet
user's ideal token delivery speed
- This tension changes dynamically as input lengths and QoE parameters vary
(c) Token-level preemption overhead:
- Fine-grained scheduling introduces overhead that may degrade throughput
- Must modulate decision frequency and account for preemption/restart costs
Scheduling approach:
- Continuously monitors attained QoE and resource usage per request
- Dynamically prioritizes requests at risk of QoE degradation
- Accounts for both memory and compute constraints
- Incorporates preemption and restart overhead into scheduling decisions
2.4 The Token Pacer
On the client side, Andes introduces a token pacer that temporarily buffers
excess tokens and delivers them at the user's consumption speed [Liu et al.,
Andes, §5]. This is crucial because:
- Server generates tokens much faster than users consume them (e.g., 11
tokens/sec vs. 4.8 tokens/sec reading speed)
- The gap between generation and consumption speed creates scheduling slack
- Pacing decouples server-side scheduling from client-side experience
- Enables the server to preempt and resume requests without the user noticing
Example [Liu et al., Andes, §2.3]: Under moderate load, the server
generates 11 tokens/sec while users consume 4.8 tokens/sec. Each request only
needs to be served for 1 second every 2.3 seconds. This means the server can
potentially serve 2.3× more requests concurrently.
2.5 Push-Based Streaming
Andes's server implements push-based streaming — immediately transmitting
tokens to the client as they are generated [Liu et al., Andes, §1]. This is
because:
- The server is resource-constrained
- Generated tokens are delivered exclusively to the submitting user
- The client-side token pacer handles smoothing
2.6 Evaluation Results
Models evaluated: Popular LLMs from 3.8B to 70B parameters, including
Dense and Mixture-of-Experts architectures, Multi-Head Attention and
Grouped-Query Attention [Liu et al., Andes, §6].
Datasets: Three datasets with varying input and output sequence lengths.
Key results [Liu et al., Andes, Abstract]:
- Up to 4.7× improvement in average QoE given the same GPU resources
compared to vLLM and Sarathi-Serve
- Up to 61% GPU savings while maintaining the same high QoE
Real-world trace evaluation [Liu et al., Andes, §2.2]:
- Replayed one-hour slice of BurstGPT trace (real-world LLM serving trace)
- vLLM with FCFS scheduling: average TTFT of 10.4 seconds during surges
(likely beyond user patience)
- vLLM average TDS: 11.2 tokens/sec (far exceeds any reading speed)
- Andes fixes both by redistributing compute across requests
2.7 Relationship to Existing Systems
Andes was evaluated against [Liu et al., Andes, §6]:
- vLLM [Kwon et al., 2023] — PagedAttention, continuous batching, FCFS
- Sarathi-Serve [Agrawal et al., 2024] — Chunked-prefill scheduling
Both systems use FCFS or simple priority policies that cause head-of-line
blocking during load surges. Andes improves on them by:
- Replacing FCFS with QoE-aware preemptive scheduling
- Operating at token granularity instead of request granularity
- Co-designing server and client (token pacer)
3. Energy-Aware LLM Serving: The ML.ENERGY Benchmark
3.1 Motivation: The Energy Bottleneck
As generative AI adoption grows explosively, energy has emerged as a critical
bottleneck resource [Chung*, Liu* et al., ML.ENERGY, §1]. Key facts:
- ML inference accounts for 80–90% of total compute demand for AI services
[Chung*, Liu* et al., ML.ENERGY, §1]
- GPUs account for 50–70% of total provisioned power in datacenters
[Chung*, Liu* et al., ML.ENERGY, §2.1]
- Getting access to energy is becoming "increasingly costly, slow, and
sometimes impossible" [Chung*, Liu* et al., ML.ENERGY, §1]
- Left unaddressed, the energy bottleneck will "hinder AI research and
development progress" and "lead to energy being squeezed out of existing
electricity grids" [Chung*, Liu* et al., ML.ENERGY, §1]
The gap: Despite its importance, energy remains a secondary consideration
compared to traditional optimization objectives like time and accuracy.
ML.ENERGY was created to bridge this gap.
3.2 The ML.ENERGY Initiative
The ML.ENERGY Benchmark is an open-source benchmark suite for measuring
inference energy consumption under realistic service environments, with a
corresponding Leaderboard at https://ml.energy/leaderboard
[Chung*, Liu* et al., ML.ENERGY, §1].
Scope as of early 2025:
- 40 different generative AI model architectures across 6 different tasks
- Tasks include: LLM chat, LLM coding, VLM visual chat, text-to-image,
text-to-video, and image-to-video generation
- Built on the Zeus library for programmatic energy measurement
[You et al., 2023]
3.3 Four Design Principles
The benchmark is guided by four core principles [Chung*, Liu* et al.,
ML.ENERGY, §2]:
Principle 1: Generalizability and Portability
- Focus on software-based GPU energy measurement
- GPUs are more standardized than other hardware components
- GPUs allow accurate software-based energy measurement (via NVML/Zeus)
- Measurements are portable across systems using the same GPU model
Principle 2: Representing Real-World Deployments
- Use production-grade software (e.g., vLLM) and hardware (e.g., NVIDIA
H100 GPUs)
- Run with workloads representative of real-world use cases
- Measure steady-state energy consumption (when batch size is saturated)
to approximate long-term deployment behavior
Principle 3: Energy Measurement at the Right Granularity
- Report energy at the per-request level (whole response), not per-token
or per-batch
- Per-request granularity captures model verbosity differences and task-
specific output length distributions
- Per-token is too fine (misses higher-level insights); per-batch is too
coarse (limited runtime insight)
Principle 4: Actionable Measurement Results
- Provide automated optimization recommendations, not just measurements
- Construct time–energy Pareto frontier across configurations
- Given user-specified latency constraints, recommend energy-optimal
configuration and expected savings
3.4 Energy Accounting Methodology
For Diffusion models (simpler case) [Chung*, Liu* et al., ML.ENERGY,
§3.2]:
Energy_request = Energy_batch / B
where B = batch size.
For LLM text generation (more complex due to iteration-level batching)
[Chung*, Liu* et al., ML.ENERGY, §3.2]:
- Submit all requests; identify the steady state (batch size saturated at
server's maximum configured batch size)
- Compute average energy per token during steady state
- Multiply by average output tokens to get per-request energy:
Energy_request = (Energy_steady / Tokens_steady) × (1/N) Σ Tokens_request,i
Why steady state matters: During ramp-up/ramp-down, the server runs with
smaller batch sizes and doesn't exhibit the same energy amortization benefits.
Steady-state measurements represent well-utilized long-term deployment.
3.5 Automated Optimization Recommendations
The benchmark constructs the time–energy Pareto frontier across all
measured configurations [Chung*, Liu* et al., ML.ENERGY, §3.3]:
- Sweep configurations (GPU model, parallelism, batch size)
- Measure time and energy for each configuration
- Build Pareto frontier (no configuration has both lower energy AND lower time)
- Given user's latency target, select minimum-energy configuration that
satisfies the constraint
Latency targets are task-specific:
- LLM chat: Average TPOT should be at least as fast as user's
reading/listening speed (connects back to QoE framework!)
- LLM coding: Total generation time for complete code snippet
- Image generation: Total generation time for complete image
- Video generation: Total generation time for complete video
3.6 Key Findings (Early 2025)
Automated optimization can lead to >40% energy savings without changing
what is computed by the model [Chung*, Liu* et al., ML.ENERGY, Abstract].
Batch size is a critical configuration that significantly affects both
generation time and energy consumption [Chung*, Liu* et al., ML.ENERGY, §3.2].
Sweeping batch size captures varying utilization levels and reveals the
time–energy trade-off.
Energy varies significantly across:
- Different model architectures for the same task
- Different tasks for the same model
- Different hardware/software configurations for the same model+task
3.7 Extensibility
The ML.ENERGY Benchmark is designed to be easily extensible [Chung*, Liu* et
al., ML.ENERGY, §3.4]:
- Already-supported model families can be extended via fine-tuned variants
- New model architectures can be added by implementing a standard interface
- Custom task datasets can be plugged in
- Open-source on GitHub: https://github.com/ml-energy/benchmark
4. Cross-Cutting Themes
4.1 The User-Centric Paradigm Shift
Jiachen Liu's work represents a fundamental paradigm shift in how we think
about LLM serving:
| Traditional Serving | User-Centric Serving (Liu's approach) |
|---|
| Optimize tokens/sec throughput | Optimize user-perceived QoE |
| TTFT + average TPOT as metrics | Full token delivery timeline as metric |
| FCFS scheduling | QoE-aware preemptive scheduling |
| Server-centric resource allocation | User consumption speed-aware allocation |
| Treat all tokens equally | "Every token matters" |
| Energy as afterthought | Energy as first-class metric (ML.ENERGY) |
4.2 The QoE–Energy Nexus
There is a natural connection between QoE and energy:
- Over-provisioning wastes energy: Generating tokens faster than users can
consume them wastes GPU cycles and energy. QoE-aware scheduling (Andes)
naturally reduces energy waste by serving only at the pace users need.
- Energy-optimal configurations for serving: The ML.ENERGY benchmark shows
that for chat applications, the latency target should be set to the user's
reading/listening speed — exactly the same insight as Andes's QoE framework.
- Batch size as a joint knob: Larger batch sizes increase throughput but
slow per-request generation and increase energy. The optimal batch size
balances QoE and energy.
4.3 The Streaming Service Analogy
Both Andes and ML.ENERGY treat LLM serving as a streaming service problem:
- Video streaming: Well-studied QoE metrics (startup delay, rebuffering,
resolution adaptation). Mature optimization landscape.
- Text streaming (LLM): Andes brings the same user-experience-first
thinking to LLM serving. The token pacer is analogous to video buffering.
- Energy streaming: ML.ENERGY treats energy as a resource that flows
through the serving pipeline, measurable at per-request granularity.
4.4 From Federated Learning to LLM Serving
Jiachen Liu's earlier work on Venn [Liu et al., MLSys 2025] addresses
resource management for collaborative learning jobs at the edge. The throughline:
- Venn: Multi-resource contention among federated learning jobs → efficient
edge resource management
- Andes: Multi-user contention for GPU resources → QoE-aware scheduling
- ML.ENERGY: Energy as a contended resource → energy-aware optimization
All three share the philosophy: understand the actual resource demands and
user needs, then optimize holistically rather than for single metrics.
5. The Broader LLM Serving Ecosystem
5.1 Key Systems for Context
To understand Andes's contributions, know the landscape:
| System | Key Innovation | Year | Reference |
|---|
| vLLM | PagedAttention for KV-cache management; continuous batching | 2023 | Kwon et al., SOSP 2023 |
| Sarathi-Serve | Chunked-prefill for better throughput | 2024 | Agrawal et al., OSDI 2024 |
| TensorRT-LLM | NVIDIA's optimized inference engine | 2023 | NVIDIA |
| SGLang | RadixAttention for prefix caching | 2024 | Zheng et al. |
| DeepSpeed-FastGen | SplitFuse for dynamic split of prompts/generations | 2024 | Microsoft |
| Andes | QoE-aware preemptive scheduling at token granularity | 2024 | Liu et al. |
| DistServe | Disaggregated prefill and decode | 2024 | Zhong et al., OSDI 2024 |
5.2 Key LLM Serving Metrics
Standard metrics used in the field:
- TTFT (Time-to-First-Token): Time from request submission to first token
delivery. Andes notes this misses everything after the first token.
- TPOT (Time-Per-Output-Token): Time between consecutive token deliveries.
Average/P90/P99 variants exist. Andes notes averages miss outlier pauses.
- TDS (Token Delivery Speed): Tokens delivered per second to the user.
Andes distinguishes server generation speed from user consumption speed.
- Throughput: Aggregate tokens/sec across all requests. Server-centric;
ignores individual request experience.
- SLO Attainment: Fraction of requests meeting Service Level Objectives.
Binary pass/fail; doesn't capture degree of degradation.
- QoE (Andes): Holistic metric capturing the full token delivery timeline
per request, normalized to [0, 1].
5.3 Key Techniques in Modern Serving
- Continuous Batching (Iteration-level Batching): Unlike static batching,
new requests can join a batch mid-execution. Standard in vLLM, Sarathi-Serve.
[Yu et al., 2022]
- PagedAttention: Virtual memory-inspired KV-cache management. Enables
memory sharing and reduces fragmentation. [Kwon et al., 2023]
- Chunked Prefill: Splitting prompt processing into chunks to reduce TTFT
and improve scheduling flexibility. [Agrawal et al., 2024]
- Speculative Decoding: Using a smaller draft model to predict tokens,
verified in parallel by the larger model. Reduces latency.
[Leviathan et al., 2023; Chen et al., 2023]
- Disaggregated Prefill-Decode: Separating prefill (prompt processing) and
decode (token generation) onto different hardware. [Zhong et al., 2024]
- Prefix Caching: Caching KV-cache for common prefixes across requests.
[Zheng et al., 2024]
6. How to Discuss LLM Serving as Jiachen Liu
6.1 Core Talking Points
When discussing LLM serving from Jiachen's perspective, lead with these ideas:
-
"The metric is wrong." Most serving systems optimize for throughput or
simple latency percentiles. These don't capture what users actually
experience. QoE is the right metric.
-
"Every token matters." User experience is determined by the entire
token delivery timeline, not just the first or last token. A long pause
in the middle of streaming degrades QoE even if TTFT and average TPOT
look fine.
-
"Users can't read faster than they read." Delivering tokens faster than
the user's reading speed doesn't improve experience. This excess capacity
is wasted — it's a resource management bug, not a feature.
-
"Energy is not optional." Energy has become a critical bottleneck for
AI infrastructure. The ML.ENERGY benchmark makes energy a first-class
metric alongside time and accuracy.
-
"User-centric serving is more efficient, not less." Andes achieves up
to 4.7× better QoE OR 61% GPU savings. Being user-centric is not at odds
with efficiency — it enables it.
6.2 Framing Conversations
When someone mentions throughput optimization:
"That's important, but it's a server-centric view. The question is whether
that throughput translates to good user experience. A system that generates
100K tokens/sec but leaves users waiting 10 seconds for their first token
has a QoE problem."
When someone asks about TTFT/TPOT:
"These are necessary but insufficient. TTFT only captures the first moment.
Average TPOT can hide long pauses. What we need is a metric that captures
the entire token delivery timeline — that's what QoE does."
When energy comes up:
"Energy is the next bottleneck. GPUs already account for 50-70% of
datacenter power. Our ML.ENERGY benchmark shows that automated optimization
can save over 40% energy without changing the model or the output. The key
insight: for chat applications, you only need to generate tokens as fast as
users can read them — anything faster wastes energy."
When discussing the future of LLM serving:
"The field is moving from server-centric to user-centric optimization. The
next frontier is jointly optimizing QoE, energy, and cost — treating the
user's experience as the primary objective while minimizing resource
consumption."
6.3 Technical Depth Signals
To demonstrate deep domain knowledge, reference:
- The cascading delay property of Andes's QoE formula — delays in earlier
tokens inflate the delay of all subsequent tokens, naturally captured by
S_delay
- The token pacer as a client-side mechanism that creates scheduling slack
by buffering server-side over-generation
- The steady-state energy accounting methodology of ML.ENERGY — why
ramp-up/ramp-down periods must be excluded from measurements
- The time–energy Pareto frontier construction for automated optimization
recommendations
- The connection between user consumption speed (QoE) and energy-optimal
batch size (ML.ENERGY) — both set the latency target at reading speed
6.4 Common Questions and Answers
Q: How does QoE relate to SLOs?
A: SLO attainment is binary (met/not met). QoE is continuous in [0, 1] — it
captures how well the experience was, not just whether a threshold was met.
A request with QoE=0.95 is meaningfully better than one with QoE=0.6, even
if both technically meet an SLO.
Q: Why preempt at token granularity instead of request granularity?
A: Request-granularity preemption wastes the tokens already generated.
Token-granularity preemption allows pausing a request mid-generation and
switching to another, redistributing GPU cycles where they improve QoE most.
The overhead is manageable with careful scheduling.
Q: How does Andes handle unpredictable response lengths?
A: The scheduler continuously monitors each request's progress and QoE state.
It doesn't need to predict response length — it dynamically reprioritizes
based on current QoE risk. Requests approaching QoE degradation get higher
priority.
Q: Is the QoE formula specific to text?
A: The formula is general to any streaming delivery with a known consumption
speed. It could apply to synthesized speech, video frames, or any incremental
delivery. The paper focuses on text streaming where reading/listening speed
provides the consumption speed parameter.
7. Key Papers — Full Bibliography
-
[Liu et al., 2024] Jiachen Liu, Jae-Won Chung, Zhiyu Wu, Fan Lai,
Myungjin Lee, Mosharaf Chowdhury. "Andes: Defining and Enhancing
Quality-of-Experience in LLM-Based Text Streaming Services." arXiv
preprint arXiv:2404.16283, 2024.
https://arxiv.org/abs/2404.16283
-
[Chung, Liu et al., 2025]** Jae-Won Chung*, Jeff J. Ma*, Ruofan Wu*,
Jiachen Liu*, Oh Jun Kweon, Yuxuan Xia, Zhiyu Wu, Mosharaf Chowdhury.
"The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement
and Optimization." NeurIPS 2025 (Spotlight). arXiv:2505.06371.
https://arxiv.org/abs/2505.06371
-
[Liu et al., 2024b] Xinjian Luo, Jiachen Liu, et al. "Efficient
Large Language Models: A Survey." Transactions on Machine Learning
Research (TMLR), 2024. arXiv:2312.03863.
https://arxiv.org/abs/2312.03863
-
[Liu et al., 2025] Jiachen Liu, Ding Ding, Fan Lai, Yiwen Zhang,
Mosharaf Chowdhury. "Venn: Resource Management for Collaborative
Learning Jobs." MLSys 2025.
-
[Kwon et al., 2023] Woosuk Kwon, Zhuohan Li, Sicheng Zhuang, Ying
Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang,
Ion Stoica. "Efficient Memory Management for Large Language Model Serving
with PagedAttention." SOSP 2023.
-
[Agrawal et al., 2024] Amey Agrawal, Nitin Kedia, Ashish Panwar,
Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Alexey Tumanov,
Ramachandran Ramjee. "Taming Throughput-Latency Tradeoff in LLM
Inference with Sarathi-Serve." OSDI 2024.
-
[Zhong et al., 2024] Yinmin Zhong, Shengyu Liu, Junda Chen,
Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, Hao Zhang. "DistServe:
Disaggregating Prefill and Decoding for Goodput-optimized Large
Language Model Serving." OSDI 2024.
-
[Yu et al., 2022] Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim,
Soojeong Kim, Byung-Gon Chun. "Orca: A Distributed Serving System for
Transformer-Based Generative Models." OSDI 2022.
-
[You et al., 2023] Jie You, Jae-Won Chung, Mosharaf Chowdhury.
"Zeus: Understanding and Optimizing GPU Energy Consumption of DNN
Training." NSDI 2023.
8. Glossary
| Term | Definition |
|---|
| QoE | Quality of Experience — holistic metric for user-perceived quality of LLM text streaming, defined as 1 - S_delay/S_whole |
| TTFT | Time-to-First-Token — latency from request submission to first token delivery |
| TPOT | Time-Per-Output-Token — latency between consecutive token deliveries |
| TDS | Token Delivery Speed — tokens delivered per second to the user |
| S_delay | Cumulative delay across all tokens: Σ(T_i^Actual - T_i^Ideal) |
| S_whole | Normalizer for QoE: Σ(T_n^Actual - T_i^Ideal) |
| Ideal Consumption Timeline | The user's expected experience — tokens arrive at reading/listening speed |
| Actual Consumption Timeline | What actually happens — tokens arrive based on server scheduling |
| Token Pacer | Client-side component that buffers excess tokens and delivers at user's consumption speed |
| Preemptive Scheduling | Scheduling that can pause mid-request to switch to another request at token granularity |
| Steady State | Period when batch size is saturated — representative of long-term deployment energy |
| Pareto Frontier | Set of configurations where no other configuration has both lower energy and lower time |
| Continuous Batching | Batching technique where new requests can join a batch mid-execution (aka iteration-level batching) |
| PagedAttention | Virtual memory-inspired KV-cache management technique from vLLM |
| Zeus | Library for programmatic GPU energy measurement, foundation of ML.ENERGY |
| Energy-per-Token | Average GPU energy consumed per generated token during steady state |
| BurstGPT | Real-world LLM serving request trace used in Andes evaluation |
9. Project Resources
