Despite the success of Large Language Models (LLMs) on various tasks following human
instructions, controlling model generation to follow strict constraints at inference time poses …

Language Model Programs, ie sophisticated pipelines of modular language model (LM)
calls, are increasingly advancing NLP tasks, but they require crafting prompts that are jointly …

Language model pretraining with next token prediction has proved effective for scaling
compute but is limited to the amount of available training data. Scaling reinforcement …

Large Language Models (LLMs) are wildly popular today and it is important to serve them
efficiently. Existing LLM serving systems are stateless across requests. Consequently, when …

Efficient management of GPU memory is essential for high throughput LLM inference. Prior
systems used to reserve KV-cache memory ahead-of-time that resulted in wasted capacity …

Online LLM inference powers many exciting applications such as intelligent chatbots and
autonomous agents. Modern LLM inference engines widely rely on request batching to …

The ability of Large Language Models (LLMs) to generate structured outputs, such as JSON,
is crucial for their use in Compound AI Systems. However, evaluating and improving this …

Simulating high quality user behavior data has always been a fundamental yet challenging
problem in human-centered applications such as recommendation systems, social networks …

With the development of large language models (LLMs), the ability to handle longer contexts
has become a key capability for Web applications such as cross-document understanding …

NP-hard problem-solving traditionally relies on heuristics, but manually crafting effective
heuristics for complex problems remains challenging. While recent work like FunSearch has …