Self-correction is a highly desirable capability of large language models (LLMs), yet it has
consistently been found to be largely ineffective in modern LLMs. Current methods for …

Direct Preference Optimization (DPO) is a widely used offline preference optimization
algorithm that reparameterizes reward functions in reinforcement learning from human …

A promising approach for improving reasoning in large language models is to use process
reward models (PRMs). PRMs provide feedback at each step of a multi-step reasoning trace …

Large language models (LLMs) are restricted to reason in the" language space", where they
typically express the reasoning process with a chain-of-thought (CoT) to solve a complex …

Iterative preference optimization methods have recently been shown to perform well for
general instruction tuning tasks, but typically make little improvement on reasoning tasks …

In order for AI systems to communicate effectively with people, they must understand how we
make decisions. However, people's decisions are not always rational, so the implicit internal …

Inference-time alignment enhances the performance of large language models without
requiring additional training or fine-tuning but presents challenges due to balancing …

Self-critic has become a crucial mechanism for enhancing the reasoning performance of
LLMs. However, current approaches mainly involve basic prompts for intuitive instance-level …

In this paper, we study format biases in reinforcement learning from human feedback
(RLHF). We observe that many widely-used preference models, including human …

OpenAI o1 represents a significant milestone in Artificial Inteiligence, which achieves expert-
level performances on many challanging tasks that require strong reasoning ability. OpenAI …