Margin Dynamics for Large Language Model Alignment
arXiv:2611.21766v1 Announce Type: new Abstract: Practitioners choose among LLM alignment methods largely by trial and error. We provide a theoretical foundation: each pairwise method defines a vector field on the probability simplex that governs how the model redistributes mass between preferred and dispreferred completions. Projecting onto a single preferred/dispreferred pair reduces this to a one-dimensional margin ODE with a closed-form solution. These solutions reveal that DPO margins grow without bound (logarithmically as $O(\log t)$ in the gradient-flow limit) while IPO margins converge
阅读原文 · arXiv 机器学习本文由 SoccerDream AI 自动聚合索引,内容版权归原作者与 arXiv 机器学习 所有。
本站仅作技术动态聚合,不构成任何形式的建议。
本站仅作技术动态聚合,不构成任何形式的建议。