Preference tuning of large language models (LLMs) relies on high-quality human preference
data, which is often expensive and time-consuming to gather. While existing methods can …