Fundamental Limits of Game-Theoretic LLM Alignment: Smith Consistency and Preference Matching
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915306048323584 |
|---|---|
| author | Shi, Zhekun Liu, Kaizhao Long, Qi Su, Weijie J. Xiao, Jiancong |
| author_facet | Shi, Zhekun Liu, Kaizhao Long, Qi Su, Weijie J. Xiao, Jiancong |
| contents | Nash Learning from Human Feedback is a game-theoretic framework for aligning large language models (LLMs) with human preferences by modeling learning as a two-player zero-sum game. However, using raw preference as the payoff in the game highly limits the potential of the game-theoretic LLM alignment framework. In this paper, we systematically study using what choices of payoff based on the pairwise human preferences can yield desirable alignment properties. We establish necessary and sufficient conditions for Condorcet consistency, diversity through mixed strategies, and Smith consistency. These results provide a theoretical foundation for the robustness of game-theoretic LLM alignment. Further, we show the impossibility of preference matching -- i.e., no smooth and learnable mappings of pairwise preferences can guarantee a unique Nash equilibrium that matches a target policy, even under standard assumptions like the Bradley-Terry-Luce model. This result highlights the fundamental limitation of game-theoretic LLM alignment. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_20627 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Fundamental Limits of Game-Theoretic LLM Alignment: Smith Consistency and Preference Matching Shi, Zhekun Liu, Kaizhao Long, Qi Su, Weijie J. Xiao, Jiancong Computer Science and Game Theory Machine Learning Nash Learning from Human Feedback is a game-theoretic framework for aligning large language models (LLMs) with human preferences by modeling learning as a two-player zero-sum game. However, using raw preference as the payoff in the game highly limits the potential of the game-theoretic LLM alignment framework. In this paper, we systematically study using what choices of payoff based on the pairwise human preferences can yield desirable alignment properties. We establish necessary and sufficient conditions for Condorcet consistency, diversity through mixed strategies, and Smith consistency. These results provide a theoretical foundation for the robustness of game-theoretic LLM alignment. Further, we show the impossibility of preference matching -- i.e., no smooth and learnable mappings of pairwise preferences can guarantee a unique Nash equilibrium that matches a target policy, even under standard assumptions like the Bradley-Terry-Luce model. This result highlights the fundamental limitation of game-theoretic LLM alignment. |
| title | Fundamental Limits of Game-Theoretic LLM Alignment: Smith Consistency and Preference Matching |
| topic | Computer Science and Game Theory Machine Learning |
| url | https://arxiv.org/abs/2505.20627 |