We present Self-Play Preference Optimization (SPO), an algorithm for reinforcement
learning from human feedback. Our approach is minimalist in that it does not require training …

Reinforcement learning from human feedback (RLHF) is a key driver of quality and safety in
state-of-the-art large language models. Yet, a surprisingly simple and strong inference-time …

We present the workflow of Online Iterative Reinforcement Learning from Human Feedback
(RLHF) in this technical report, which is widely reported to outperform its offline counterpart …

Reverse-Kullback-Leibler (KL) regularization has emerged to be a predominant technique
used to enhance policy optimization in reinforcement learning (RL) and reinforcement …

Self-supervision has the potential to transform reinforcement learning (RL), paralleling the
breakthroughs it has enabled in other areas of machine learning. While self-supervised …

Reinforcement Learning from Human Feedback (RLHF), the standard for aligning Large
Language Models (LLMs) with human values, is known to fail to satisfy properties that are …

Preference alignment in Large Language Models (LLMs) has significantly improved their
ability to adhere to human instructions and intentions. However, existing direct alignment …

This dissertation investigates several challenges in artificial intelligence (AI) alignment and
reinforcement learning (RL), particularly focusing on applications when only preference …

Self-supervision has the potential to transform reinforcement learning (RL), paralleling the
breakthroughs it has enabled in other areas of machine learning. While self-supervised …