AlignDiff addresses the challenge of low-quality preference data impacting large language model alignment. The framework utilizes intrinsic model signals to identify and prioritize preference samples. Specifically, AlignDiff employs both positive and inverse preference signals to initially select samples exhibiting clear preferences. Following this, it focuses on samples with the largest average negative log-likelihood gap, encouraging the model to learn from more difficult instances. The framework was evaluated on LLaMA and Qwen model families, alongside AlpacaEval 2.0, Arena-Hard, and MT-Bench benchmarks. Results consistently outperformed seven strong baselines across all settings. Comprehensive ablation studies validated the framework’s effectiveness, and further experiments showed that difficulty-based curriculum learning enhances model performance.
Source: https://arxiv.org/abs/2609.05899