Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2504.15431 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910916202725376 |
|---|---|
| author | Han, Sungjun Suk, Juyoung An, Suyeong Kim, Hyungguk Kim, Kyuseok Yang, Wonsuk Choi, Seungtaek Shin, Jamin |
| author_facet | Han, Sungjun Suk, Juyoung An, Suyeong Kim, Hyungguk Kim, Kyuseok Yang, Wonsuk Choi, Seungtaek Shin, Jamin |
| contents | We introduce Trillion-7B, the most token-efficient Korean-centric multilingual LLM available. Our novel Cross-lingual Document Attention (XLDA) mechanism enables highly efficient and effective knowledge transfer from English to target languages like Korean and Japanese. Combined with optimized data mixtures, language-specific filtering, and tailored tokenizer construction, Trillion-7B achieves competitive performance while dedicating only 10\% of its 2T training tokens to multilingual data and requiring just 59.4K H100 GPU hours (\$148K) for full training. Comprehensive evaluations across 27 benchmarks in four languages demonstrate Trillion-7B's robust multilingual performance and exceptional cross-lingual consistency. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_15431 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Trillion 7B Technical Report Han, Sungjun Suk, Juyoung An, Suyeong Kim, Hyungguk Kim, Kyuseok Yang, Wonsuk Choi, Seungtaek Shin, Jamin Computation and Language Artificial Intelligence Machine Learning We introduce Trillion-7B, the most token-efficient Korean-centric multilingual LLM available. Our novel Cross-lingual Document Attention (XLDA) mechanism enables highly efficient and effective knowledge transfer from English to target languages like Korean and Japanese. Combined with optimized data mixtures, language-specific filtering, and tailored tokenizer construction, Trillion-7B achieves competitive performance while dedicating only 10\% of its 2T training tokens to multilingual data and requiring just 59.4K H100 GPU hours (\$148K) for full training. Comprehensive evaluations across 27 benchmarks in four languages demonstrate Trillion-7B's robust multilingual performance and exceptional cross-lingual consistency. |
| title | Trillion 7B Technical Report |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2504.15431 |