Can AI Assistants Know What They Don't Know?
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866929225961832448 |
|---|---|
| author | Cheng, Qinyuan Sun, Tianxiang Liu, Xiangyang Zhang, Wenwei Yin, Zhangyue Li, Shimin Li, Linyang He, Zhengfu Chen, Kai Qiu, Xipeng |
| author_facet | Cheng, Qinyuan Sun, Tianxiang Liu, Xiangyang Zhang, Wenwei Yin, Zhangyue Li, Shimin Li, Linyang He, Zhengfu Chen, Kai Qiu, Xipeng |
| contents | Recently, AI assistants based on large language models (LLMs) show surprising performance in many tasks, such as dialogue, solving math problems, writing code, and using tools. Although LLMs possess intensive world knowledge, they still make factual errors when facing some knowledge intensive tasks, like open-domain question answering. These untruthful responses from the AI assistant may cause significant risks in practical applications. We believe that an AI assistant's refusal to answer questions it does not know is a crucial method for reducing hallucinations and making the assistant truthful. Therefore, in this paper, we ask the question "Can AI assistants know what they don't know and express them through natural language?" To answer this question, we construct a model-specific "I don't know" (Idk) dataset for an assistant, which contains its known and unknown questions, based on existing open-domain question answering datasets. Then we align the assistant with its corresponding Idk dataset and observe whether it can refuse to answer its unknown questions after alignment. Experimental results show that after alignment with Idk datasets, the assistant can refuse to answer most its unknown questions. For questions they attempt to answer, the accuracy is significantly higher than before the alignment. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_13275 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Can AI Assistants Know What They Don't Know? Cheng, Qinyuan Sun, Tianxiang Liu, Xiangyang Zhang, Wenwei Yin, Zhangyue Li, Shimin Li, Linyang He, Zhengfu Chen, Kai Qiu, Xipeng Computation and Language Artificial Intelligence Recently, AI assistants based on large language models (LLMs) show surprising performance in many tasks, such as dialogue, solving math problems, writing code, and using tools. Although LLMs possess intensive world knowledge, they still make factual errors when facing some knowledge intensive tasks, like open-domain question answering. These untruthful responses from the AI assistant may cause significant risks in practical applications. We believe that an AI assistant's refusal to answer questions it does not know is a crucial method for reducing hallucinations and making the assistant truthful. Therefore, in this paper, we ask the question "Can AI assistants know what they don't know and express them through natural language?" To answer this question, we construct a model-specific "I don't know" (Idk) dataset for an assistant, which contains its known and unknown questions, based on existing open-domain question answering datasets. Then we align the assistant with its corresponding Idk dataset and observe whether it can refuse to answer its unknown questions after alignment. Experimental results show that after alignment with Idk datasets, the assistant can refuse to answer most its unknown questions. For questions they attempt to answer, the accuracy is significantly higher than before the alignment. |
| title | Can AI Assistants Know What They Don't Know? |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2401.13275 |