TimelyFreeze: Adaptive Parameter Freezing Mechanism for Pipeline Parallelism
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914310433800192 |
|---|---|
| author | Cho, Seonghye Han, Jaemin Kim, Hyunjin Jung, Euisoo Lee, Jae-Gil |
| author_facet | Cho, Seonghye Han, Jaemin Kim, Hyunjin Jung, Euisoo Lee, Jae-Gil |
| contents | Pipeline parallelism enables training models that exceed single-device memory, but practical throughput remains limited by pipeline bubbles. Although parameter freezing can improve training throughput by adaptively skipping backward computation, existing methods often over-freeze parameters, resulting in unnecessary accuracy degradation. To address this issue, we propose TimelyFreeze, which models the pipeline schedule as a directed acyclic graph and solves a linear program to compute optimal freeze ratios that minimize batch execution time under accuracy constraints. Experiments show that TimelyFreeze achieves up to 40% training throughput improvement on LLaMA-8B with comparable accuracy. Overall, it enables faster large-scale model training without compromising convergence and generalizes across diverse pipeline-parallel settings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_05754 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | TimelyFreeze: Adaptive Parameter Freezing Mechanism for Pipeline Parallelism Cho, Seonghye Han, Jaemin Kim, Hyunjin Jung, Euisoo Lee, Jae-Gil Distributed, Parallel, and Cluster Computing Artificial Intelligence Pipeline parallelism enables training models that exceed single-device memory, but practical throughput remains limited by pipeline bubbles. Although parameter freezing can improve training throughput by adaptively skipping backward computation, existing methods often over-freeze parameters, resulting in unnecessary accuracy degradation. To address this issue, we propose TimelyFreeze, which models the pipeline schedule as a directed acyclic graph and solves a linear program to compute optimal freeze ratios that minimize batch execution time under accuracy constraints. Experiments show that TimelyFreeze achieves up to 40% training throughput improvement on LLaMA-8B with comparable accuracy. Overall, it enables faster large-scale model training without compromising convergence and generalizes across diverse pipeline-parallel settings. |
| title | TimelyFreeze: Adaptive Parameter Freezing Mechanism for Pipeline Parallelism |
| topic | Distributed, Parallel, and Cluster Computing Artificial Intelligence |
| url | https://arxiv.org/abs/2602.05754 |