Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | https://arxiv.org/abs/2504.15630 |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909588436025344 |
|---|---|
| author | Yuan, Xiaowei Yang, Zhao Huang, Ziyang Wang, Yequan Fan, Siqi Ju, Yiming Zhao, Jun Liu, Kang |
| author_facet | Yuan, Xiaowei Yang, Zhao Huang, Ziyang Wang, Yequan Fan, Siqi Ju, Yiming Zhao, Jun Liu, Kang |
| contents | Large Language Models (LLMs) have demonstrated remarkable capabilities in various tasks, yet they often struggle with context-faithfulness generations that properly reflect contextual knowledge. While existing approaches focus on enhancing the decoding strategies, they ignore the fundamental mechanism of how contextual information is processed within LLMs' internal states. As a result, LLMs remain limited in their ability to fully leverage contextual knowledge. In this paper, we propose Context-aware Layer Enhancement (CaLE), a novel intervention method that enhances the utilization of contextual knowledge within LLMs' internal representations. By employing V-usable information analysis, CaLE strategically amplifies the growth of contextual information at an optimal layer, thereby enriching representations in the final layer. Our experiments demonstrate that CaLE effectively improves context-faithful generation in Question-Answering tasks, particularly in scenarios involving unknown or conflicting contextual knowledge. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_15630 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Exploiting Contextual Knowledge in LLMs through V-usable Information based Layer Enhancement Yuan, Xiaowei Yang, Zhao Huang, Ziyang Wang, Yequan Fan, Siqi Ju, Yiming Zhao, Jun Liu, Kang Computation and Language Large Language Models (LLMs) have demonstrated remarkable capabilities in various tasks, yet they often struggle with context-faithfulness generations that properly reflect contextual knowledge. While existing approaches focus on enhancing the decoding strategies, they ignore the fundamental mechanism of how contextual information is processed within LLMs' internal states. As a result, LLMs remain limited in their ability to fully leverage contextual knowledge. In this paper, we propose Context-aware Layer Enhancement (CaLE), a novel intervention method that enhances the utilization of contextual knowledge within LLMs' internal representations. By employing V-usable information analysis, CaLE strategically amplifies the growth of contextual information at an optimal layer, thereby enriching representations in the final layer. Our experiments demonstrate that CaLE effectively improves context-faithful generation in Question-Answering tasks, particularly in scenarios involving unknown or conflicting contextual knowledge. |
| title | Exploiting Contextual Knowledge in LLMs through V-usable Information based Layer Enhancement |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2504.15630 |