Have We Designed Generalizable Structural Knowledge Promptings? Systematic Evaluation and Rethinking
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910768916594688 |
|---|---|
| author | Zhang, Yichi Chen, Zhuo Guo, Lingbing Xu, Yajing Chen, Shaokai Sun, Mengshu Hu, Binbin Zhang, Zhiqiang Liang, Lei Zhang, Wen Chen, Huajun |
| author_facet | Zhang, Yichi Chen, Zhuo Guo, Lingbing Xu, Yajing Chen, Shaokai Sun, Mengshu Hu, Binbin Zhang, Zhiqiang Liang, Lei Zhang, Wen Chen, Huajun |
| contents | Large language models (LLMs) have demonstrated exceptional performance in text generation within current NLP research. However, the lack of factual accuracy is still a dark cloud hanging over the LLM skyscraper. Structural knowledge prompting (SKP) is a prominent paradigm to integrate external knowledge into LLMs by incorporating structural representations, achieving state-of-the-art results in many knowledge-intensive tasks. However, existing methods often focus on specific problems, lacking a comprehensive exploration of the generalization and capability boundaries of SKP. This paper aims to evaluate and rethink the generalization capability of the SKP paradigm from four perspectives including Granularity, Transferability, Scalability, and Universality. To provide a thorough evaluation, we introduce a novel multi-granular, multi-level benchmark called SUBARU, consisting of 9 different tasks with varying levels of granularity and difficulty. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_00244 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Have We Designed Generalizable Structural Knowledge Promptings? Systematic Evaluation and Rethinking Zhang, Yichi Chen, Zhuo Guo, Lingbing Xu, Yajing Chen, Shaokai Sun, Mengshu Hu, Binbin Zhang, Zhiqiang Liang, Lei Zhang, Wen Chen, Huajun Computation and Language Large language models (LLMs) have demonstrated exceptional performance in text generation within current NLP research. However, the lack of factual accuracy is still a dark cloud hanging over the LLM skyscraper. Structural knowledge prompting (SKP) is a prominent paradigm to integrate external knowledge into LLMs by incorporating structural representations, achieving state-of-the-art results in many knowledge-intensive tasks. However, existing methods often focus on specific problems, lacking a comprehensive exploration of the generalization and capability boundaries of SKP. This paper aims to evaluate and rethink the generalization capability of the SKP paradigm from four perspectives including Granularity, Transferability, Scalability, and Universality. To provide a thorough evaluation, we introduce a novel multi-granular, multi-level benchmark called SUBARU, consisting of 9 different tasks with varying levels of granularity and difficulty. |
| title | Have We Designed Generalizable Structural Knowledge Promptings? Systematic Evaluation and Rethinking |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2501.00244 |