Document Details

Clip: Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences Corby Rosset ∗ Ching-An Cheng Arindam Mitra Michael Santacroce Ahmed Awadallah ∗ Tengyang Xie ∗ Microsoft Research Abstract This paper studies post-training large language models (LLMs) using preference feedback from a powerful oracle to help a model iteratively improve over itself. The typical approach for post-training LLMs involves Reinforcement Learning from Human Feedback (RLHF), which traditionally separates reward learning and subsequent policy opti- mization. However, such a reward maximization approach is limited by the nature of “point-wise” rewards (such as that

Filename: 2404.03715.pdf

Notes: