ALLaM: Large Language Models for Arabic and English
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909263972007936 |
|---|---|
| author | Bari, M Saiful Alnumay, Yazeed Alzahrani, Norah A. Alotaibi, Nouf M. Alyahya, Hisham A. AlRashed, Sultan Mirza, Faisal A. Alsubaie, Shaykhah Z. Alahmed, Hassan A. Alabduljabbar, Ghadah Alkhathran, Raghad Almushayqih, Yousef Alnajim, Raneem Alsubaihi, Salman Mansour, Maryam Al Alrubaian, Majed Alammari, Ali Alawami, Zaki Al-Thubaity, Abdulmohsen Abdelali, Ahmed Kuriakose, Jeril Abujabal, Abdalghani Al-Twairesh, Nora Alowisheq, Areeb Khan, Haidar |
| author_facet | Bari, M Saiful Alnumay, Yazeed Alzahrani, Norah A. Alotaibi, Nouf M. Alyahya, Hisham A. AlRashed, Sultan Mirza, Faisal A. Alsubaie, Shaykhah Z. Alahmed, Hassan A. Alabduljabbar, Ghadah Alkhathran, Raghad Almushayqih, Yousef Alnajim, Raneem Alsubaihi, Salman Mansour, Maryam Al Alrubaian, Majed Alammari, Ali Alawami, Zaki Al-Thubaity, Abdulmohsen Abdelali, Ahmed Kuriakose, Jeril Abujabal, Abdalghani Al-Twairesh, Nora Alowisheq, Areeb Khan, Haidar |
| contents | We present ALLaM: Arabic Large Language Model, a series of large language models to support the ecosystem of Arabic Language Technologies (ALT). ALLaM is carefully trained considering the values of language alignment and knowledge transfer at scale. Our autoregressive decoder-only architecture models demonstrate how second-language acquisition via vocabulary expansion and pretraining on a mixture of Arabic and English text can steer a model towards a new language (Arabic) without any catastrophic forgetting in the original language (English). Furthermore, we highlight the effectiveness of using parallel/translated data to aid the process of knowledge alignment between languages. Finally, we show that extensive alignment with human preferences can significantly enhance the performance of a language model compared to models of a larger scale with lower quality alignment. ALLaM achieves state-of-the-art performance in various Arabic benchmarks, including MMLU Arabic, ACVA, and Arabic Exams. Our aligned models improve both in Arabic and English from their base aligned models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_15390 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ALLaM: Large Language Models for Arabic and English Bari, M Saiful Alnumay, Yazeed Alzahrani, Norah A. Alotaibi, Nouf M. Alyahya, Hisham A. AlRashed, Sultan Mirza, Faisal A. Alsubaie, Shaykhah Z. Alahmed, Hassan A. Alabduljabbar, Ghadah Alkhathran, Raghad Almushayqih, Yousef Alnajim, Raneem Alsubaihi, Salman Mansour, Maryam Al Alrubaian, Majed Alammari, Ali Alawami, Zaki Al-Thubaity, Abdulmohsen Abdelali, Ahmed Kuriakose, Jeril Abujabal, Abdalghani Al-Twairesh, Nora Alowisheq, Areeb Khan, Haidar Computation and Language Artificial Intelligence We present ALLaM: Arabic Large Language Model, a series of large language models to support the ecosystem of Arabic Language Technologies (ALT). ALLaM is carefully trained considering the values of language alignment and knowledge transfer at scale. Our autoregressive decoder-only architecture models demonstrate how second-language acquisition via vocabulary expansion and pretraining on a mixture of Arabic and English text can steer a model towards a new language (Arabic) without any catastrophic forgetting in the original language (English). Furthermore, we highlight the effectiveness of using parallel/translated data to aid the process of knowledge alignment between languages. Finally, we show that extensive alignment with human preferences can significantly enhance the performance of a language model compared to models of a larger scale with lower quality alignment. ALLaM achieves state-of-the-art performance in various Arabic benchmarks, including MMLU Arabic, ACVA, and Arabic Exams. Our aligned models improve both in Arabic and English from their base aligned models. |
| title | ALLaM: Large Language Models for Arabic and English |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2407.15390 |