Expanding Foundational Language Capabilities in Open-Source LLMs through a Korean Case Study
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866918135693574144 |
|---|---|
| author | Lim, Junghwan Jo, Gangwon Lee, Sungmin Park, Jiyoung Kim, Dongseok Kim, Jihwan Lee, Junhyeok Cheung, Wai Ting Choi, Dahye Choi, Kibong Huh, Jaeyeon Kim, Beomgyu Kim, Jangwoong Kim, Taehyun Lee, Haesol Lee, Jeesoo Oh, Dongpin Song, Changseok Suh, Daewon |
| author_facet | Lim, Junghwan Jo, Gangwon Lee, Sungmin Park, Jiyoung Kim, Dongseok Kim, Jihwan Lee, Junhyeok Cheung, Wai Ting Choi, Dahye Choi, Kibong Huh, Jaeyeon Kim, Beomgyu Kim, Jangwoong Kim, Taehyun Lee, Haesol Lee, Jeesoo Oh, Dongpin Song, Changseok Suh, Daewon |
| contents | We introduce Llama-3-Motif, a language model consisting of 102 billion parameters, specifically designed to enhance Korean capabilities while retaining strong performance in English. Developed on the Llama 3 architecture, Llama-3-Motif employs advanced training techniques, including LlamaPro and Masked Structure Growth, to effectively scale the model without altering its core Transformer architecture. Using the MoAI platform for efficient training across hyperscale GPU clusters, we optimized Llama-3-Motif using a carefully curated dataset that maintains a balanced ratio of Korean and English data. Llama-3-Motif shows decent performance on Korean-specific benchmarks, outperforming existing models and achieving results comparable to GPT-4. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_03972 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Expanding Foundational Language Capabilities in Open-Source LLMs through a Korean Case Study Lim, Junghwan Jo, Gangwon Lee, Sungmin Park, Jiyoung Kim, Dongseok Kim, Jihwan Lee, Junhyeok Cheung, Wai Ting Choi, Dahye Choi, Kibong Huh, Jaeyeon Kim, Beomgyu Kim, Jangwoong Kim, Taehyun Lee, Haesol Lee, Jeesoo Oh, Dongpin Song, Changseok Suh, Daewon Computation and Language Artificial Intelligence Machine Learning We introduce Llama-3-Motif, a language model consisting of 102 billion parameters, specifically designed to enhance Korean capabilities while retaining strong performance in English. Developed on the Llama 3 architecture, Llama-3-Motif employs advanced training techniques, including LlamaPro and Masked Structure Growth, to effectively scale the model without altering its core Transformer architecture. Using the MoAI platform for efficient training across hyperscale GPU clusters, we optimized Llama-3-Motif using a carefully curated dataset that maintains a balanced ratio of Korean and English data. Llama-3-Motif shows decent performance on Korean-specific benchmarks, outperforming existing models and achieving results comparable to GPT-4. |
| title | Expanding Foundational Language Capabilities in Open-Source LLMs through a Korean Case Study |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2509.03972 |