Large language models (LLMs) are increasingly used for complex tasks that require multiple
generation calls, advanced prompting techniques, control flow, and structured …

Large language models (LLMs) are increasingly used for complex tasks that require multiple
generation calls, advanced prompting techniques, control flow, and structured …

The increasing usage of Large Language Models (LLMs) has resulted in a surging demand
for planet-scale serving systems, where tens of thousands of GPUs continuously serve …

As large language models (LLMs) evolve to handle increasingly longer contexts, serving
inference requests for context lengths in the range of millions of tokens presents unique …

Each LLM serving request goes through two phases. The first is prefill which processes the
entire input prompt and produces the first output token and the second is decode which …

Each request in LLM inference goes through two phases: compute-bound prefill and
memory-bandwidth-bound decode. To improve GPU utilization, recent systems use hybrid …

This paper introduces AdaServe, the first LLM serving system to support SLO customization
through fine-grained speculative decoding. AdaServe leverages the logits of a draft model to …

Scaling-law has guided the language model designing for past years, however, it is worth
noting that the scaling laws of NLP cannot be directly applied to RecSys due to the following …

Large Language Models (LLMs) are widely used in today's tasks of natural language
processing. To support applications like multi-turn chats, document understanding, and …

The rapid growth of generative AI and its integration into everyday workflows have
significantly increased the demand for large language model (LLM) inference services …