AceGPT, Localizing Large Language Models in Arabic
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909156417470464 |
|---|---|
| author | Huang, Huang Yu, Fei Zhu, Jianqing Sun, Xuening Cheng, Hao Song, Dingjie Chen, Zhihong Alharthi, Abdulmohsen An, Bang He, Juncai Liu, Ziche Zhang, Zhiyi Chen, Junying Li, Jianquan Wang, Benyou Zhang, Lian Sun, Ruoyu Wan, Xiang Li, Haizhou Xu, Jinchao |
| author_facet | Huang, Huang Yu, Fei Zhu, Jianqing Sun, Xuening Cheng, Hao Song, Dingjie Chen, Zhihong Alharthi, Abdulmohsen An, Bang He, Juncai Liu, Ziche Zhang, Zhiyi Chen, Junying Li, Jianquan Wang, Benyou Zhang, Lian Sun, Ruoyu Wan, Xiang Li, Haizhou Xu, Jinchao |
| contents | This paper is devoted to the development of a localized Large Language Model (LLM) specifically for Arabic, a language imbued with unique cultural characteristics inadequately addressed by current mainstream models. Significant concerns emerge when addressing cultural sensitivity and local values. To address this, the paper proposes a comprehensive solution that includes further pre-training with Arabic texts, Supervised Fine-Tuning (SFT) utilizing native Arabic instructions, and GPT-4 responses in Arabic, alongside Reinforcement Learning with AI Feedback (RLAIF) employing a reward model attuned to local culture and values. The goal is to cultivate culturally cognizant and value-aligned Arabic LLMs capable of accommodating the diverse, application-specific needs of Arabic-speaking communities.
Comprehensive evaluations reveal that the resulting model, dubbed `AceGPT', sets the state-of-the-art standard for open Arabic LLMs across various benchmarks. Codes, data, and models are in https://github.com/FreedomIntelligence/AceGPT. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2309_12053 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | AceGPT, Localizing Large Language Models in Arabic Huang, Huang Yu, Fei Zhu, Jianqing Sun, Xuening Cheng, Hao Song, Dingjie Chen, Zhihong Alharthi, Abdulmohsen An, Bang He, Juncai Liu, Ziche Zhang, Zhiyi Chen, Junying Li, Jianquan Wang, Benyou Zhang, Lian Sun, Ruoyu Wan, Xiang Li, Haizhou Xu, Jinchao Computation and Language This paper is devoted to the development of a localized Large Language Model (LLM) specifically for Arabic, a language imbued with unique cultural characteristics inadequately addressed by current mainstream models. Significant concerns emerge when addressing cultural sensitivity and local values. To address this, the paper proposes a comprehensive solution that includes further pre-training with Arabic texts, Supervised Fine-Tuning (SFT) utilizing native Arabic instructions, and GPT-4 responses in Arabic, alongside Reinforcement Learning with AI Feedback (RLAIF) employing a reward model attuned to local culture and values. The goal is to cultivate culturally cognizant and value-aligned Arabic LLMs capable of accommodating the diverse, application-specific needs of Arabic-speaking communities. Comprehensive evaluations reveal that the resulting model, dubbed `AceGPT', sets the state-of-the-art standard for open Arabic LLMs across various benchmarks. Codes, data, and models are in https://github.com/FreedomIntelligence/AceGPT. |
| title | AceGPT, Localizing Large Language Models in Arabic |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2309.12053 |