IP-Dialog: Evaluating Implicit Personalization in Dialogue Systems with Synthetic Data
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912411000242176 |
|---|---|
| author | Peng, Bo Wang, Zhiheng Gong, Heyang Lu, Chaochao |
| author_facet | Peng, Bo Wang, Zhiheng Gong, Heyang Lu, Chaochao |
| contents | In modern dialogue systems, the ability to implicitly infer user backgrounds from conversations and leverage this information for personalized assistance is crucial. However, the scarcity of high-quality data remains a fundamental challenge to evaluating and improving this capability. Traditional dataset construction methods are labor-intensive, resource-demanding, and raise privacy concerns. To address these issues, we propose a novel approach for automatic synthetic data generation and introduce the Implicit Personalized Dialogue (IP-Dialog) benchmark along with a training dataset, covering 10 tasks and 12 user attribute types. Additionally, we develop a systematic evaluation framework with four metrics to assess both attribute awareness and reasoning capabilities. We further propose five causal graphs to elucidate models' reasoning pathways during implicit personalization. Extensive experiments yield insightful observations and prove the reliability of our dataset. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_02449 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | IP-Dialog: Evaluating Implicit Personalization in Dialogue Systems with Synthetic Data Peng, Bo Wang, Zhiheng Gong, Heyang Lu, Chaochao Computation and Language Human-Computer Interaction In modern dialogue systems, the ability to implicitly infer user backgrounds from conversations and leverage this information for personalized assistance is crucial. However, the scarcity of high-quality data remains a fundamental challenge to evaluating and improving this capability. Traditional dataset construction methods are labor-intensive, resource-demanding, and raise privacy concerns. To address these issues, we propose a novel approach for automatic synthetic data generation and introduce the Implicit Personalized Dialogue (IP-Dialog) benchmark along with a training dataset, covering 10 tasks and 12 user attribute types. Additionally, we develop a systematic evaluation framework with four metrics to assess both attribute awareness and reasoning capabilities. We further propose five causal graphs to elucidate models' reasoning pathways during implicit personalization. Extensive experiments yield insightful observations and prove the reliability of our dataset. |
| title | IP-Dialog: Evaluating Implicit Personalization in Dialogue Systems with Synthetic Data |
| topic | Computation and Language Human-Computer Interaction |
| url | https://arxiv.org/abs/2506.02449 |