Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2409.11726 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908371608666112 |
|---|---|
| author | Zhang, Wenyuan Nie, Shuaiyi Sheng, Jiawei Zhang, Zefeng Zhang, Xinghua He, Yongquan Liu, Tingwen |
| author_facet | Zhang, Wenyuan Nie, Shuaiyi Sheng, Jiawei Zhang, Zefeng Zhang, Xinghua He, Yongquan Liu, Tingwen |
| contents | Large language model (LLM) role-playing has gained widespread attention. Authentic character knowledge is crucial for constructing realistic LLM role-playing agents. However, existing works usually overlook the exploration of LLMs' ability to detect characters' known knowledge errors (KKE) and unknown knowledge errors (UKE) while playing roles, which would lead to low-quality automatic construction of character trainable corpus. In this paper, we propose RoleKE-Bench to evaluate LLMs' ability to detect errors in KKE and UKE. The results indicate that even the latest LLMs struggle to detect these two types of errors effectively, especially when it comes to familiar knowledge. We experimented with various reasoning strategies and propose an agent-based reasoning method, Self-Recollection and Self-Doubt (S$^2$RD), to explore further the potential for improving error detection capabilities. Experiments show that our method effectively improves the LLMs' ability to detect error character knowledge, but it remains an issue that requires ongoing attention. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_11726 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Revealing and Mitigating the Challenge of Detecting Character Knowledge Errors in LLM Role-Playing Zhang, Wenyuan Nie, Shuaiyi Sheng, Jiawei Zhang, Zefeng Zhang, Xinghua He, Yongquan Liu, Tingwen Computation and Language Human-Computer Interaction Large language model (LLM) role-playing has gained widespread attention. Authentic character knowledge is crucial for constructing realistic LLM role-playing agents. However, existing works usually overlook the exploration of LLMs' ability to detect characters' known knowledge errors (KKE) and unknown knowledge errors (UKE) while playing roles, which would lead to low-quality automatic construction of character trainable corpus. In this paper, we propose RoleKE-Bench to evaluate LLMs' ability to detect errors in KKE and UKE. The results indicate that even the latest LLMs struggle to detect these two types of errors effectively, especially when it comes to familiar knowledge. We experimented with various reasoning strategies and propose an agent-based reasoning method, Self-Recollection and Self-Doubt (S$^2$RD), to explore further the potential for improving error detection capabilities. Experiments show that our method effectively improves the LLMs' ability to detect error character knowledge, but it remains an issue that requires ongoing attention. |
| title | Revealing and Mitigating the Challenge of Detecting Character Knowledge Errors in LLM Role-Playing |
| topic | Computation and Language Human-Computer Interaction |
| url | https://arxiv.org/abs/2409.11726 |