Aligning VLM Assistants with Personalized Situated Cognition
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916771946037248 |
|---|---|
| author | Li, Yongqi Zhou, Shen Li, Xiaohu Miao, Xin Wen, Jintao Xu, Mayi Chen, Jianhao Pan, Birong Kang, Hankun Zhu, Yuanyuan Zhong, Ming Qian, Tieyun |
| author_facet | Li, Yongqi Zhou, Shen Li, Xiaohu Miao, Xin Wen, Jintao Xu, Mayi Chen, Jianhao Pan, Birong Kang, Hankun Zhu, Yuanyuan Zhong, Ming Qian, Tieyun |
| contents | Vision-language models (VLMs) aligned with general human objectives, such as being harmless and hallucination-free, have become valuable assistants of humans in managing visual tasks. However, people with diversified backgrounds have different cognition even in the same situation. Consequently, they may have personalized expectations for VLM assistants. This highlights the urgent need to align VLM assistants with personalized situated cognition for real-world assistance. To study this problem, we first simplify it by characterizing individuals based on the sociological concept of Role-Set. Then, we propose to evaluate the individuals' actions to examine whether the personalized alignment is achieved. Further, we construct a benchmark named PCogAlignBench, which includes 18k instances and 20 individuals with different Role-Sets. Finally, we present a framework called PCogAlign, which constructs a cognition-aware and action-based reward model for personalized alignment. Experimental results and human evaluations demonstrate the reliability of the PCogAlignBench and the effectiveness of our proposed PCogAlign. We will open-source the constructed benchmark and code at https://github.com/NLPGM/PCogAlign. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_00930 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Aligning VLM Assistants with Personalized Situated Cognition Li, Yongqi Zhou, Shen Li, Xiaohu Miao, Xin Wen, Jintao Xu, Mayi Chen, Jianhao Pan, Birong Kang, Hankun Zhu, Yuanyuan Zhong, Ming Qian, Tieyun Artificial Intelligence Computation and Language Vision-language models (VLMs) aligned with general human objectives, such as being harmless and hallucination-free, have become valuable assistants of humans in managing visual tasks. However, people with diversified backgrounds have different cognition even in the same situation. Consequently, they may have personalized expectations for VLM assistants. This highlights the urgent need to align VLM assistants with personalized situated cognition for real-world assistance. To study this problem, we first simplify it by characterizing individuals based on the sociological concept of Role-Set. Then, we propose to evaluate the individuals' actions to examine whether the personalized alignment is achieved. Further, we construct a benchmark named PCogAlignBench, which includes 18k instances and 20 individuals with different Role-Sets. Finally, we present a framework called PCogAlign, which constructs a cognition-aware and action-based reward model for personalized alignment. Experimental results and human evaluations demonstrate the reliability of the PCogAlignBench and the effectiveness of our proposed PCogAlign. We will open-source the constructed benchmark and code at https://github.com/NLPGM/PCogAlign. |
| title | Aligning VLM Assistants with Personalized Situated Cognition |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2506.00930 |