PersianRAG: A Retrieval-Augmented Generation System for Persian Language
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916470001238016 |
|---|---|
| author | Hosseini, Hossein Zare, Mohammad Sobhan Mohammadi, Amir Hossein Kazemi, Arefeh Zojaji, Zahra Nematbakhsh, Mohammad Ali |
| author_facet | Hosseini, Hossein Zare, Mohammad Sobhan Mohammadi, Amir Hossein Kazemi, Arefeh Zojaji, Zahra Nematbakhsh, Mohammad Ali |
| contents | Retrieval augmented generation (RAG) models, which integrate large-scale pre-trained generative models with external retrieval mechanisms, have shown significant success in various natural language processing (NLP) tasks. However, applying RAG models in Persian language as a low-resource language, poses distinct challenges. These challenges primarily involve the preprocessing, embedding, retrieval, prompt construction, language modeling, and response evaluation of the system. In this paper, we address the challenges towards implementing a real-world RAG system for Persian language called PersianRAG. We propose novel solutions to overcome these obstacles and evaluate our approach using several Persian benchmark datasets. Our experimental results demonstrate the capability of the PersianRAG framework to enhance question answering task in Persian. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_02832 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | PersianRAG: A Retrieval-Augmented Generation System for Persian Language Hosseini, Hossein Zare, Mohammad Sobhan Mohammadi, Amir Hossein Kazemi, Arefeh Zojaji, Zahra Nematbakhsh, Mohammad Ali Computation and Language Artificial Intelligence Information Retrieval Retrieval augmented generation (RAG) models, which integrate large-scale pre-trained generative models with external retrieval mechanisms, have shown significant success in various natural language processing (NLP) tasks. However, applying RAG models in Persian language as a low-resource language, poses distinct challenges. These challenges primarily involve the preprocessing, embedding, retrieval, prompt construction, language modeling, and response evaluation of the system. In this paper, we address the challenges towards implementing a real-world RAG system for Persian language called PersianRAG. We propose novel solutions to overcome these obstacles and evaluate our approach using several Persian benchmark datasets. Our experimental results demonstrate the capability of the PersianRAG framework to enhance question answering task in Persian. |
| title | PersianRAG: A Retrieval-Augmented Generation System for Persian Language |
| topic | Computation and Language Artificial Intelligence Information Retrieval |
| url | https://arxiv.org/abs/2411.02832 |