A Tale of Trust and Accuracy: Base vs. Instruct LLMs in RAG Systems
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914843773108224 |
|---|---|
| author | Cuconasu, Florin Trappolini, Giovanni Tonellotto, Nicola Silvestri, Fabrizio |
| author_facet | Cuconasu, Florin Trappolini, Giovanni Tonellotto, Nicola Silvestri, Fabrizio |
| contents | Retrieval Augmented Generation (RAG) represents a significant advancement in artificial intelligence combining a retrieval phase with a generative phase, with the latter typically being powered by large language models (LLMs). The current common practices in RAG involve using "instructed" LLMs, which are fine-tuned with supervised training to enhance their ability to follow instructions and are aligned with human preferences using state-of-the-art techniques. Contrary to popular belief, our study demonstrates that base models outperform their instructed counterparts in RAG tasks by 20% on average under our experimental settings. This finding challenges the prevailing assumptions about the superiority of instructed LLMs in RAG applications. Further investigations reveal a more nuanced situation, questioning fundamental aspects of RAG and suggesting the need for broader discussions on the topic; or, as Fromm would have it, "Seldom is a glance at the statistics enough to understand the meaning of the figures". |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_14972 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | A Tale of Trust and Accuracy: Base vs. Instruct LLMs in RAG Systems Cuconasu, Florin Trappolini, Giovanni Tonellotto, Nicola Silvestri, Fabrizio Computation and Language Information Retrieval Retrieval Augmented Generation (RAG) represents a significant advancement in artificial intelligence combining a retrieval phase with a generative phase, with the latter typically being powered by large language models (LLMs). The current common practices in RAG involve using "instructed" LLMs, which are fine-tuned with supervised training to enhance their ability to follow instructions and are aligned with human preferences using state-of-the-art techniques. Contrary to popular belief, our study demonstrates that base models outperform their instructed counterparts in RAG tasks by 20% on average under our experimental settings. This finding challenges the prevailing assumptions about the superiority of instructed LLMs in RAG applications. Further investigations reveal a more nuanced situation, questioning fundamental aspects of RAG and suggesting the need for broader discussions on the topic; or, as Fromm would have it, "Seldom is a glance at the statistics enough to understand the meaning of the figures". |
| title | A Tale of Trust and Accuracy: Base vs. Instruct LLMs in RAG Systems |
| topic | Computation and Language Information Retrieval |
| url | https://arxiv.org/abs/2406.14972 |