AlpsBench: An LLM Personalization Benchmark for Real-Dialogue Memorization and Preference Alignment
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911671458463744 |
|---|---|
| author | Xiao, Jianfei Yu, Xiang Wang, Chengbing Zheng, Wuqiang Lin, Xinyu Liu, Kaining Ding, Hongxun Zhang, Yang Wang, Wenjie Feng, Fuli He, Xiangnan |
| author_facet | Xiao, Jianfei Yu, Xiang Wang, Chengbing Zheng, Wuqiang Lin, Xinyu Liu, Kaining Ding, Hongxun Zhang, Yang Wang, Wenjie Feng, Fuli He, Xiangnan |
| contents | As Large Language Models (LLMs) evolve into lifelong AI assistants, LLM personalization has become a critical frontier. However, progress is currently bottlenecked by the absence of a gold-standard evaluation benchmark. Existing benchmarks either overlook personalized information management that is critical for personalization or rely heavily on synthetic dialogues, which exhibit an inherent distribution gap from real-world dialogue. To bridge this gap, we introduce AlpsBench, An LLM PerSonalization benchmark derived from real-world human-LLM dialogues. AlpsBench comprises 2,500 long-term interaction sequences curated from WildChat, paired with human-verified structured memories that encapsulate both explicit and implicit personalization signals. We define four pivotal tasks - personalized information extraction, updating, retrieval, and utilization - and establish protocols to evaluate the entire lifecycle of memory management. Our benchmarking of frontier LLMs and memory-centric systems reveals that: (i) models struggle to reliably extract latent user traits; (ii) memory updating faces a performance ceiling even in the strongest models; (iii) retrieval accuracy declines sharply in the presence of large distractor pools; and (iv) while explicit memory mechanisms improve recall, they do not inherently guarantee more preference-aligned or emotionally resonant responses. AlpsBench aims to provide a comprehensive framework. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_26680 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | AlpsBench: An LLM Personalization Benchmark for Real-Dialogue Memorization and Preference Alignment Xiao, Jianfei Yu, Xiang Wang, Chengbing Zheng, Wuqiang Lin, Xinyu Liu, Kaining Ding, Hongxun Zhang, Yang Wang, Wenjie Feng, Fuli He, Xiangnan Computation and Language Artificial Intelligence As Large Language Models (LLMs) evolve into lifelong AI assistants, LLM personalization has become a critical frontier. However, progress is currently bottlenecked by the absence of a gold-standard evaluation benchmark. Existing benchmarks either overlook personalized information management that is critical for personalization or rely heavily on synthetic dialogues, which exhibit an inherent distribution gap from real-world dialogue. To bridge this gap, we introduce AlpsBench, An LLM PerSonalization benchmark derived from real-world human-LLM dialogues. AlpsBench comprises 2,500 long-term interaction sequences curated from WildChat, paired with human-verified structured memories that encapsulate both explicit and implicit personalization signals. We define four pivotal tasks - personalized information extraction, updating, retrieval, and utilization - and establish protocols to evaluate the entire lifecycle of memory management. Our benchmarking of frontier LLMs and memory-centric systems reveals that: (i) models struggle to reliably extract latent user traits; (ii) memory updating faces a performance ceiling even in the strongest models; (iii) retrieval accuracy declines sharply in the presence of large distractor pools; and (iv) while explicit memory mechanisms improve recall, they do not inherently guarantee more preference-aligned or emotionally resonant responses. AlpsBench aims to provide a comprehensive framework. |
| title | AlpsBench: An LLM Personalization Benchmark for Real-Dialogue Memorization and Preference Alignment |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2603.26680 |