OrthAlign: Orthogonal Subspace Decomposition for Non-Interfering Multi-Objective Alignment
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916979023020032 |
|---|---|
| author | Lin, Liang Xu, Zhihao Dong, Junhao Zhao, Jian Yuan, Yuchen Zhang, Guibin Yu, Miao Zhang, Yiming Yao, Zhengtao Yi, Huahui Liu, Dongrui Li, Xinfeng Wang, Kun |
| author_facet | Lin, Liang Xu, Zhihao Dong, Junhao Zhao, Jian Yuan, Yuchen Zhang, Guibin Yu, Miao Zhang, Yiming Yao, Zhengtao Yi, Huahui Liu, Dongrui Li, Xinfeng Wang, Kun |
| contents | Large language model (LLM) alignment faces a critical dilemma when addressing multiple human preferences: improvements in one dimension frequently come at the expense of others, creating unavoidable trade-offs between competing objectives like helpfulness and harmlessness. While prior work mainly focuses on constraint-based optimization algorithms and data selection strategies to mitigate conflicts, these approaches overlook the fundamental issue of resolving conflicts directly at the parameter level. In this paper, we present OrthAlign, an innovative approach that pioneers a new paradigm by leveraging orthogonal subspace decomposition to fundamentally resolve gradient-level conflicts in multi-objective preference alignment. OrthAlign strategically decomposes parameter update spaces into orthogonal subspaces, ensuring that optimization toward different preferences occurs in mathematically non-interfering directions. Building upon this, we provide theoretical guarantees demonstrating that when parameter increments satisfy both orthogonal subspace constraints and spectral norm bounds, the resulting updates exhibit linear Lipschitz growth rather than exponential instability, ensuring stable convergence across all preference dimensions. Extensive experiments show that: I. OrthAlign achieves maximum single-preference improvements ranging from 34.61% to 50.89% after multiple-objective alignment across helpful, harmless, and truthful dimensions. II. With an average overall reward improvement of 13.96%. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_24610 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | OrthAlign: Orthogonal Subspace Decomposition for Non-Interfering Multi-Objective Alignment Lin, Liang Xu, Zhihao Dong, Junhao Zhao, Jian Yuan, Yuchen Zhang, Guibin Yu, Miao Zhang, Yiming Yao, Zhengtao Yi, Huahui Liu, Dongrui Li, Xinfeng Wang, Kun Machine Learning Computation and Language Large language model (LLM) alignment faces a critical dilemma when addressing multiple human preferences: improvements in one dimension frequently come at the expense of others, creating unavoidable trade-offs between competing objectives like helpfulness and harmlessness. While prior work mainly focuses on constraint-based optimization algorithms and data selection strategies to mitigate conflicts, these approaches overlook the fundamental issue of resolving conflicts directly at the parameter level. In this paper, we present OrthAlign, an innovative approach that pioneers a new paradigm by leveraging orthogonal subspace decomposition to fundamentally resolve gradient-level conflicts in multi-objective preference alignment. OrthAlign strategically decomposes parameter update spaces into orthogonal subspaces, ensuring that optimization toward different preferences occurs in mathematically non-interfering directions. Building upon this, we provide theoretical guarantees demonstrating that when parameter increments satisfy both orthogonal subspace constraints and spectral norm bounds, the resulting updates exhibit linear Lipschitz growth rather than exponential instability, ensuring stable convergence across all preference dimensions. Extensive experiments show that: I. OrthAlign achieves maximum single-preference improvements ranging from 34.61% to 50.89% after multiple-objective alignment across helpful, harmless, and truthful dimensions. II. With an average overall reward improvement of 13.96%. |
| title | OrthAlign: Orthogonal Subspace Decomposition for Non-Interfering Multi-Objective Alignment |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2509.24610 |