Enhancing Q&A with Domain-Specific Fine-Tuning and Iterative Reasoning: A Comparative Study
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916214870114304 |
|---|---|
| author | Nguyen, Zooey Annunziata, Anthony Luong, Vinh Dinh, Sang Le, Quynh Ha, Anh Hai Le, Chanh Phan, Hong An Raghavan, Shruti Nguyen, Christopher |
| author_facet | Nguyen, Zooey Annunziata, Anthony Luong, Vinh Dinh, Sang Le, Quynh Ha, Anh Hai Le, Chanh Phan, Hong An Raghavan, Shruti Nguyen, Christopher |
| contents | This paper investigates the impact of domain-specific model fine-tuning and of reasoning mechanisms on the performance of question-answering (Q&A) systems powered by large language models (LLMs) and Retrieval-Augmented Generation (RAG). Using the FinanceBench SEC financial filings dataset, we observe that, for RAG, combining a fine-tuned embedding model with a fine-tuned LLM achieves better accuracy than generic models, with relatively greater gains attributable to fine-tuned embedding models. Additionally, employing reasoning iterations on top of RAG delivers an even bigger jump in performance, enabling the Q&A systems to get closer to human-expert quality. We discuss the implications of such findings, propose a structured technical design space capturing major technical components of Q&A AI, and provide recommendations for making high-impact technical choices for such components. We plan to follow up on this work with actionable guides for AI teams and further investigations into the impact of domain-specific augmentation in RAG and into agentic AI capabilities such as advanced planning and reasoning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_11792 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Enhancing Q&A with Domain-Specific Fine-Tuning and Iterative Reasoning: A Comparative Study Nguyen, Zooey Annunziata, Anthony Luong, Vinh Dinh, Sang Le, Quynh Ha, Anh Hai Le, Chanh Phan, Hong An Raghavan, Shruti Nguyen, Christopher Artificial Intelligence This paper investigates the impact of domain-specific model fine-tuning and of reasoning mechanisms on the performance of question-answering (Q&A) systems powered by large language models (LLMs) and Retrieval-Augmented Generation (RAG). Using the FinanceBench SEC financial filings dataset, we observe that, for RAG, combining a fine-tuned embedding model with a fine-tuned LLM achieves better accuracy than generic models, with relatively greater gains attributable to fine-tuned embedding models. Additionally, employing reasoning iterations on top of RAG delivers an even bigger jump in performance, enabling the Q&A systems to get closer to human-expert quality. We discuss the implications of such findings, propose a structured technical design space capturing major technical components of Q&A AI, and provide recommendations for making high-impact technical choices for such components. We plan to follow up on this work with actionable guides for AI teams and further investigations into the impact of domain-specific augmentation in RAG and into agentic AI capabilities such as advanced planning and reasoning. |
| title | Enhancing Q&A with Domain-Specific Fine-Tuning and Iterative Reasoning: A Comparative Study |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2404.11792 |