EvoWiki: Evaluating LLMs on Evolving Knowledge
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910750731141120 |
|---|---|
| author | Tang, Wei Cao, Yixin Deng, Yang Ying, Jiahao Wang, Bo Yang, Yizhe Zhao, Yuyue Zhang, Qi Huang, Xuanjing Jiang, Yugang Liao, Yong |
| author_facet | Tang, Wei Cao, Yixin Deng, Yang Ying, Jiahao Wang, Bo Yang, Yizhe Zhao, Yuyue Zhang, Qi Huang, Xuanjing Jiang, Yugang Liao, Yong |
| contents | Knowledge utilization is a critical aspect of LLMs, and understanding how they adapt to evolving knowledge is essential for their effective deployment. However, existing benchmarks are predominantly static, failing to capture the evolving nature of LLMs and knowledge, leading to inaccuracies and vulnerabilities such as contamination. In this paper, we introduce EvoWiki, an evolving dataset designed to reflect knowledge evolution by categorizing information into stable, evolved, and uncharted states. EvoWiki is fully auto-updatable, enabling precise evaluation of continuously changing knowledge and newly released LLMs. Through experiments with Retrieval-Augmented Generation (RAG) and Contunual Learning (CL), we evaluate how effectively LLMs adapt to evolving knowledge. Our results indicate that current models often struggle with evolved knowledge, frequently providing outdated or incorrect responses. Moreover, the dataset highlights a synergistic effect between RAG and CL, demonstrating their potential to better adapt to evolving knowledge. EvoWiki provides a robust benchmark for advancing future research on the knowledge evolution capabilities of large language models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_13582 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | EvoWiki: Evaluating LLMs on Evolving Knowledge Tang, Wei Cao, Yixin Deng, Yang Ying, Jiahao Wang, Bo Yang, Yizhe Zhao, Yuyue Zhang, Qi Huang, Xuanjing Jiang, Yugang Liao, Yong Computation and Language Knowledge utilization is a critical aspect of LLMs, and understanding how they adapt to evolving knowledge is essential for their effective deployment. However, existing benchmarks are predominantly static, failing to capture the evolving nature of LLMs and knowledge, leading to inaccuracies and vulnerabilities such as contamination. In this paper, we introduce EvoWiki, an evolving dataset designed to reflect knowledge evolution by categorizing information into stable, evolved, and uncharted states. EvoWiki is fully auto-updatable, enabling precise evaluation of continuously changing knowledge and newly released LLMs. Through experiments with Retrieval-Augmented Generation (RAG) and Contunual Learning (CL), we evaluate how effectively LLMs adapt to evolving knowledge. Our results indicate that current models often struggle with evolved knowledge, frequently providing outdated or incorrect responses. Moreover, the dataset highlights a synergistic effect between RAG and CL, demonstrating their potential to better adapt to evolving knowledge. EvoWiki provides a robust benchmark for advancing future research on the knowledge evolution capabilities of large language models. |
| title | EvoWiki: Evaluating LLMs on Evolving Knowledge |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2412.13582 |