PLaMo-100B: A Ground-Up Language Model Designed for Japanese Proficiency
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908098872999936 |
|---|---|
| author | Elements, Preferred : Abe, Kenshin Chubachi, Kaizaburo Fujita, Yasuhiro Hirokawa, Yuta Imajo, Kentaro Kataoka, Toshiki Komatsu, Hiroyoshi Mikami, Hiroaki Mogami, Tsuguo Murai, Shogo Nakago, Kosuke Nishino, Daisuke Ogawa, Toru Okanohara, Daisuke Ozaki, Yoshihiko Sano, Shotaro Suzuki, Shuji Xu, Tianqi Yanase, Toshihiko |
| author_facet | Elements, Preferred : Abe, Kenshin Chubachi, Kaizaburo Fujita, Yasuhiro Hirokawa, Yuta Imajo, Kentaro Kataoka, Toshiki Komatsu, Hiroyoshi Mikami, Hiroaki Mogami, Tsuguo Murai, Shogo Nakago, Kosuke Nishino, Daisuke Ogawa, Toru Okanohara, Daisuke Ozaki, Yoshihiko Sano, Shotaro Suzuki, Shuji Xu, Tianqi Yanase, Toshihiko |
| contents | We introduce PLaMo-100B, a large-scale language model designed for Japanese proficiency. The model was trained from scratch using 2 trillion tokens, with architecture such as QK Normalization and Z-Loss to ensure training stability during the training process. Post-training techniques, including Supervised Fine-Tuning and Direct Preference Optimization, were applied to refine the model's performance. Benchmark evaluations suggest that PLaMo-100B performs well, particularly in Japanese-specific tasks, achieving results that are competitive with frontier models like GPT-4. The base model is available at https://huggingface.co/pfnet/plamo-100b. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_07563 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | PLaMo-100B: A Ground-Up Language Model Designed for Japanese Proficiency Elements, Preferred : Abe, Kenshin Chubachi, Kaizaburo Fujita, Yasuhiro Hirokawa, Yuta Imajo, Kentaro Kataoka, Toshiki Komatsu, Hiroyoshi Mikami, Hiroaki Mogami, Tsuguo Murai, Shogo Nakago, Kosuke Nishino, Daisuke Ogawa, Toru Okanohara, Daisuke Ozaki, Yoshihiko Sano, Shotaro Suzuki, Shuji Xu, Tianqi Yanase, Toshihiko Computation and Language Artificial Intelligence Machine Learning We introduce PLaMo-100B, a large-scale language model designed for Japanese proficiency. The model was trained from scratch using 2 trillion tokens, with architecture such as QK Normalization and Z-Loss to ensure training stability during the training process. Post-training techniques, including Supervised Fine-Tuning and Direct Preference Optimization, were applied to refine the model's performance. Benchmark evaluations suggest that PLaMo-100B performs well, particularly in Japanese-specific tasks, achieving results that are competitive with frontier models like GPT-4. The base model is available at https://huggingface.co/pfnet/plamo-100b. |
| title | PLaMo-100B: A Ground-Up Language Model Designed for Japanese Proficiency |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2410.07563 |