| _version_ | 1866901630973116416 |
|---|---|
| author | Lirov, Kharki |
| author_facet | Lirov, Kharki |
| contents | <p>We introduce Motif-Upcycling, a structure-preserving framework for adapting pretrained Transformer models. The key idea is that common feed-forward modules, including SwiGLU FFNs, can be exactly factorized along their intermediate channel axis into motif-aligned components. With neutral routing, the factorized module computes the same function as the original pretrained block at initialization. We further introduce Scale-Aware Residual Control (SARC), an identity-preserving control motif that modulates the magnitude of trainable residual interventions relative to the residual stream. We also propose Emergence as Coupled Budget Thresholds (ECBT), a conditional model showing that apparent capability cliffs can arise from multiplicatively coupled motif effectiveness curves under uniform budget allocation.</p> |
| format | Recurso digital |
| id | zenodo_https___doi_org_10_5281_zenodo_19840136 |
| institution | Zenodo |
| language | eng |
| publishDate | 2026 |
| publisher | Zenodo |
| record_format | zenodo |
| spellingShingle | Motif-Upcycling: Structure-Preserving Adaptation of Transformer Models Lirov, Kharki computer science Machine learning natural language process deep lear neural network arch <p>We introduce Motif-Upcycling, a structure-preserving framework for adapting pretrained Transformer models. The key idea is that common feed-forward modules, including SwiGLU FFNs, can be exactly factorized along their intermediate channel axis into motif-aligned components. With neutral routing, the factorized module computes the same function as the original pretrained block at initialization. We further introduce Scale-Aware Residual Control (SARC), an identity-preserving control motif that modulates the magnitude of trainable residual interventions relative to the residual stream. We also propose Emergence as Coupled Budget Thresholds (ECBT), a conditional model showing that apparent capability cliffs can arise from multiplicatively coupled motif effectiveness curves under uniform budget allocation.</p> |
| title | Motif-Upcycling: Structure-Preserving Adaptation of Transformer Models |
| topic | computer science Machine learning natural language process deep lear neural network arch |
| url | https://doi.org/10.5281/zenodo.19840136 |