Large language models are zero-shot time series forecasters
By encoding time series as a string of numerical digits, we can frame time series forecasting
as next-token prediction in text. Develo** this approach, we find that large language …
as next-token prediction in text. Develo** this approach, we find that large language …
Faith and fate: Limits of transformers on compositionality
Transformer large language models (LLMs) have sparked admiration for their exceptional
performance on tasks that demand intricate multi-step reasoning. Yet, these models …
performance on tasks that demand intricate multi-step reasoning. Yet, these models …
Weak-to-strong generalization: Eliciting strong capabilities with weak supervision
Widely used alignment techniques, such as reinforcement learning from human feedback
(RLHF), rely on the ability of humans to supervise model behavior-for example, to evaluate …
(RLHF), rely on the ability of humans to supervise model behavior-for example, to evaluate …
What can transformers learn in-context? a case study of simple function classes
In-context learning is the ability of a model to condition on a prompt sequence consisting of
in-context examples (input-output pairs corresponding to some task) along with a new query …
in-context examples (input-output pairs corresponding to some task) along with a new query …
Least-to-most prompting enables complex reasoning in large language models
Chain-of-thought prompting has demonstrated remarkable performance on various natural
language reasoning tasks. However, it tends to perform poorly on tasks which requires …
language reasoning tasks. However, it tends to perform poorly on tasks which requires …
Exploring length generalization in large language models
The ability to extrapolate from short problem instances to longer ones is an important form of
out-of-distribution generalization in reasoning tasks, and is crucial when learning from …
out-of-distribution generalization in reasoning tasks, and is crucial when learning from …
Transformers learn shortcuts to automata
Algorithmic reasoning requires capabilities which are most naturally understood through
recurrent models of computation, like the Turing machine. However, Transformer models …
recurrent models of computation, like the Turing machine. However, Transformer models …
Combinatorial optimization and reasoning with graph neural networks
Combinatorial optimization is a well-established area in operations research and computer
science. Until recently, its methods have focused on solving problem instances in isolation …
science. Until recently, its methods have focused on solving problem instances in isolation …
Easy-to-hard generalization: Scalable alignment beyond human supervision
Current AI alignment methodologies rely on human-provided demonstrations or judgments,
and the learned capabilities of AI systems would be upper-bounded by human capabilities …
and the learned capabilities of AI systems would be upper-bounded by human capabilities …
Transformers can achieve length generalization but not robustly
Length generalization, defined as the ability to extrapolate from shorter training sequences
to longer test ones, is a significant challenge for language models. This issue persists even …
to longer test ones, is a significant challenge for language models. This issue persists even …