Mainstream approaches to aligning large language models (LLMs) heavily rely on human
preference data, particularly when models require periodic updates. The standard process …