Kanana: Compute-efficient Bilingual Language Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909516173410304 |
|---|---|
| author | Kanana LLM Team Bak, Yunju Lee, Hojin Ryu, Minho Ham, Jiyeon Jung, Seungjae Nam, Daniel Wontae Eo, Taegyeong Lee, Donghun Jung, Doohae Kim, Boseop Kim, Nayeon Park, Jaesun Kim, Hyunho Ko, Hyunwoong Lee, Changmin On, Kyoung-Woon Baeg, Seulye Cho, Junrae Jung, Sunghee Kang, Jieun Kim, EungGyun Kim, Eunhwa Ko, Byeongil Lee, Daniel Lee, Minchul Lee, Miok Lee, Shinbok Seo, Gaeun |
| author_facet | Kanana LLM Team Bak, Yunju Lee, Hojin Ryu, Minho Ham, Jiyeon Jung, Seungjae Nam, Daniel Wontae Eo, Taegyeong Lee, Donghun Jung, Doohae Kim, Boseop Kim, Nayeon Park, Jaesun Kim, Hyunho Ko, Hyunwoong Lee, Changmin On, Kyoung-Woon Baeg, Seulye Cho, Junrae Jung, Sunghee Kang, Jieun Kim, EungGyun Kim, Eunhwa Ko, Byeongil Lee, Daniel Lee, Minchul Lee, Miok Lee, Shinbok Seo, Gaeun |
| contents | We introduce Kanana, a series of bilingual language models that demonstrate exceeding performance in Korean and competitive performance in English. The computational cost of Kanana is significantly lower than that of state-of-the-art models of similar size. The report details the techniques employed during pre-training to achieve compute-efficient yet competitive models, including high quality data filtering, staged pre-training, depth up-scaling, and pruning and distillation. Furthermore, the report outlines the methodologies utilized during the post-training of the Kanana models, encompassing supervised fine-tuning and preference optimization, aimed at enhancing their capability for seamless interaction with users. Lastly, the report elaborates on plausible approaches used for language model adaptation to specific scenarios, such as embedding, retrieval augmented generation, and function calling. The Kanana model series spans from 2.1B to 32.5B parameters with 2.1B models (base, instruct, embedding) publicly released to promote research on Korean language models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_18934 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Kanana: Compute-efficient Bilingual Language Models Kanana LLM Team Bak, Yunju Lee, Hojin Ryu, Minho Ham, Jiyeon Jung, Seungjae Nam, Daniel Wontae Eo, Taegyeong Lee, Donghun Jung, Doohae Kim, Boseop Kim, Nayeon Park, Jaesun Kim, Hyunho Ko, Hyunwoong Lee, Changmin On, Kyoung-Woon Baeg, Seulye Cho, Junrae Jung, Sunghee Kang, Jieun Kim, EungGyun Kim, Eunhwa Ko, Byeongil Lee, Daniel Lee, Minchul Lee, Miok Lee, Shinbok Seo, Gaeun Computation and Language Machine Learning We introduce Kanana, a series of bilingual language models that demonstrate exceeding performance in Korean and competitive performance in English. The computational cost of Kanana is significantly lower than that of state-of-the-art models of similar size. The report details the techniques employed during pre-training to achieve compute-efficient yet competitive models, including high quality data filtering, staged pre-training, depth up-scaling, and pruning and distillation. Furthermore, the report outlines the methodologies utilized during the post-training of the Kanana models, encompassing supervised fine-tuning and preference optimization, aimed at enhancing their capability for seamless interaction with users. Lastly, the report elaborates on plausible approaches used for language model adaptation to specific scenarios, such as embedding, retrieval augmented generation, and function calling. The Kanana model series spans from 2.1B to 32.5B parameters with 2.1B models (base, instruct, embedding) publicly released to promote research on Korean language models. |
| title | Kanana: Compute-efficient Bilingual Language Models |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2502.18934 |