We present Self-Play Preference Optimization (SPO), an algorithm for reinforcement
learning from human feedback. Our approach is minimalist in that it does not require training …

Reinforcement Learning from Human Feedback (RLHF) has emerged as a popular
paradigm for aligning models with human intent. Typically RLHF algorithms operate in two …

Reinforcement Learning from Human Feedback (RLHF) has been crucial to the recent
success of Large Language Models (LLMs), however, it is often a complex and brittle …

The goal of reinforcement learning (RL) is to find a policy that maximizes the expected
cumulative return. It has been shown that this objective can be represented as an …

Reinforcement Learning is a promising tool for learning complex policies even in fast-
moving and object-interactive domains where human teleoperation or hard-coded policies …

The ability to interactively learn skills from human guidance and adjust behavior according
to human preference is crucial to accelerating robot learning. But human guidance is an …

Learning from human feedback has gained traction in fields like robotics and natural
language processing in recent years. While prior works mostly rely on human feedback in …

Preference-based reinforcement learning (PbRL) aligns a robot behavior with human
preferences via a reward function learned from binary feedback over agent behaviors. We …

Offline Goal-Conditioned Reinforcement Learning (GCRL) is tasked with learning to achieve
multiple goals in an environment purely from offline datasets using sparse reward functions …

It is well known that Reinforcement Learning (RL) can be formulated as a convex program
with linear constraints. The dual form of this formulation is unconstrained, which we refer to …