Unleashing Scalable Context Parallelism for Foundation Models Pre-Training via FCP
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914546297339904 |
|---|---|
| author | Zhao, Yilong Nie, Xiaonan Zhu, Kan Ma, Shuang Lai, Zhichao Hao, Hongxiang Zhou, Yang Kasikci, Baris Stoica, Ion |
| author_facet | Zhao, Yilong Nie, Xiaonan Zhu, Kan Ma, Shuang Lai, Zhichao Hao, Hongxiang Zhou, Yang Kasikci, Baris Stoica, Ion |
| contents | Context parallelism (CP) has been widely adopted to support the growing context length in foundation model pretraining. However, existing designs fail to handle the large variation in sequence length from training datasets, resulting in suboptimal performance. These methods often over-shard short sequences, leading to compute inefficiency and excessive communication, or process long and short sequences separately without proper bin-packing, causing workload imbalance. In this paper, we propose FCP, a flexible context parallelism paradigm that shards and schedules sequences at block-level granularity. Instead of relying on rigid communication topologies such as ring, FCP enables arbitrary peer-to-peer communication, allowing flexible placement of sequence blocks across workers. By bin-packing blocks from both short and long sequences, FCP achieves both high compute efficiency and balanced workload distribution. Extensive evaluations show that FCP attains near-linear scalability on up to 256 NVIDIA GPUs, with 1.13x-2.21x improvement in the attention MFU. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_08524 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Unleashing Scalable Context Parallelism for Foundation Models Pre-Training via FCP Zhao, Yilong Nie, Xiaonan Zhu, Kan Ma, Shuang Lai, Zhichao Hao, Hongxiang Zhou, Yang Kasikci, Baris Stoica, Ion Distributed, Parallel, and Cluster Computing Context parallelism (CP) has been widely adopted to support the growing context length in foundation model pretraining. However, existing designs fail to handle the large variation in sequence length from training datasets, resulting in suboptimal performance. These methods often over-shard short sequences, leading to compute inefficiency and excessive communication, or process long and short sequences separately without proper bin-packing, causing workload imbalance. In this paper, we propose FCP, a flexible context parallelism paradigm that shards and schedules sequences at block-level granularity. Instead of relying on rigid communication topologies such as ring, FCP enables arbitrary peer-to-peer communication, allowing flexible placement of sequence blocks across workers. By bin-packing blocks from both short and long sequences, FCP achieves both high compute efficiency and balanced workload distribution. Extensive evaluations show that FCP attains near-linear scalability on up to 256 NVIDIA GPUs, with 1.13x-2.21x improvement in the attention MFU. |
| title | Unleashing Scalable Context Parallelism for Foundation Models Pre-Training via FCP |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2605.08524 |