Deterministic Continuous Replacement: Fast and Stable Module Replacement in Pretrained Transformers
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914168579293184 |
|---|---|
| author | Bradbury, Rowan Ashok, Aniket Srinivasan Kasanagottu, Sai Ram Jhingran, Gunmay Meng, Shuai |
| author_facet | Bradbury, Rowan Ashok, Aniket Srinivasan Kasanagottu, Sai Ram Jhingran, Gunmay Meng, Shuai |
| contents | Replacing modules in pretrained models, especially swapping quadratic self-attention for efficient attention alternatives, poses a hard optimization problem: cold-start reinitialization destabilizes frozen backbones. We isolate this core stability challenge in a controlled study. Deterministic Continuous Replacement (DCR) blends teacher and student outputs with a deterministic, annealed weight. Theoretically, DCR eliminates gate-induced gradient variance inherent to stochastic replacement. In a single-seed study, DCR attains faster convergence and stronger alignment than stochastic gating and distillation baselines on controlled attention replacement, establishing a foundation for heterogeneous operator swaps. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_18670 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Deterministic Continuous Replacement: Fast and Stable Module Replacement in Pretrained Transformers Bradbury, Rowan Ashok, Aniket Srinivasan Kasanagottu, Sai Ram Jhingran, Gunmay Meng, Shuai Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition Replacing modules in pretrained models, especially swapping quadratic self-attention for efficient attention alternatives, poses a hard optimization problem: cold-start reinitialization destabilizes frozen backbones. We isolate this core stability challenge in a controlled study. Deterministic Continuous Replacement (DCR) blends teacher and student outputs with a deterministic, annealed weight. Theoretically, DCR eliminates gate-induced gradient variance inherent to stochastic replacement. In a single-seed study, DCR attains faster convergence and stronger alignment than stochastic gating and distillation baselines on controlled attention replacement, establishing a foundation for heterogeneous operator swaps. |
| title | Deterministic Continuous Replacement: Fast and Stable Module Replacement in Pretrained Transformers |
| topic | Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2511.18670 |