This survey offers a comprehensive overview of recent advancements in Large Language
Model (LLM) serving systems, focusing on research since the year 2023. We specifically …

Large language models (LLMs) are increasingly integrated into many online services, yet
they remain cost-prohibitive to deploy due to the requirement of expensive GPU instances …

As large language models (LLMs) take on complex tasks, their inputs are supplemented with
longer contexts that incorporate domain knowledge. Yet using long contexts is challenging …

Large language model (LLM) serving is becoming an increasingly critical workload for cloud
providers. Existing LLM serving systems focus on interactive requests, such as chatbots and …

Large Language Model (LLM) workloads have distinct prefill and decode phases with
different compute and memory requirements which should ideally be accounted for when …

Social interactions are fundamental to human life. The recent emergence of large language
models (LLMs)-based virtual assistants has demonstrated their potential to revolutionize …

In Large Language Model (LLM) inference, the output length of an LLM request is typically
regarded as not known a priori. Consequently, most LLM serving systems employ a simple …

This paper introduces AdaServe, the first LLM serving system to support SLO customization
through fine-grained speculative decoding. AdaServe leverages the logits of a draft model to …

We propose TETRIS, a novel method that optimizes the total throughput of batch speculative
decoding in multi-request settings. Unlike existing methods that optimize for a single request …

Large language models (LLMs) offer powerful capabilities but come with significant
environmental costs, particularly in carbon emissions. Existing studies benchmark these …