Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2505.22296 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911198093508608 |
|---|---|
| author | Zou, Haosheng Lv, Xiaowei Jia, Shousheng Li, Lin Gong, Xiaochun Zhang, Xiangzheng |
| author_facet | Zou, Haosheng Lv, Xiaowei Jia, Shousheng Li, Lin Gong, Xiaochun Zhang, Xiangzheng |
| contents | Adding sequence parallelism into LLaMA-Factory, we open-sourced 360-LLaMA-Factory at https://github.com/Qihoo360/360-LLaMA-Factory. 360-LLaMA-Factory has received wide recognition and used in models such as Light-R1 arXiv:2503.10460, TinyR1 arXiv:2503.04872, Kaggle AIMO math models and also in large companies' training frameworks. This technical report delves deeper into the different sequence parallel modes behind 360-LLaMA-Factory and discusses our implementation insights. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_22296 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | 360-LLaMA-Factory: Plug & Play Sequence Parallelism for Long Post-Training Zou, Haosheng Lv, Xiaowei Jia, Shousheng Li, Lin Gong, Xiaochun Zhang, Xiangzheng Computation and Language Machine Learning Adding sequence parallelism into LLaMA-Factory, we open-sourced 360-LLaMA-Factory at https://github.com/Qihoo360/360-LLaMA-Factory. 360-LLaMA-Factory has received wide recognition and used in models such as Light-R1 arXiv:2503.10460, TinyR1 arXiv:2503.04872, Kaggle AIMO math models and also in large companies' training frameworks. This technical report delves deeper into the different sequence parallel modes behind 360-LLaMA-Factory and discusses our implementation insights. |
| title | 360-LLaMA-Factory: Plug & Play Sequence Parallelism for Long Post-Training |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2505.22296 |