SALAAD: Sparse And Low-Rank Adaptation via ADMM for Large Language Model Inference
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913171597426688 |
|---|---|
| author | Ma, Hao Bal, Melis Ilayda Zhang, Liang Li, Bingcong He, Niao Zeilinger, Melanie Muehlebach, Michael |
| author_facet | Ma, Hao Bal, Melis Ilayda Zhang, Liang Li, Bingcong He, Niao Zeilinger, Melanie Muehlebach, Michael |
| contents | Modern large language models are increasingly deployed under compute and memory constraints, making flexible control of model capacity a central challenge. While sparse and low-rank structures naturally trade off capacity and performance, existing approaches often rely on heuristic designs that ignore layer and matrix heterogeneity or require model-specific architectural modifications. We propose SALAAD, a plug-and-play framework applicable to different model architectures that induces sparse and low-rank structures during training. By formulating structured weight learning under an augmented Lagrangian framework and introducing an adaptive controller that dynamically balances the training loss and structural constraints, SALAAD preserves the stability of standard training dynamics while enabling explicit control over the evolution of effective model capacity during training. Experiments across model scales show that SALAAD substantially reduces memory consumption during deployment while achieving performance comparable to ad-hoc methods. Moreover, a single training run yields a continuous spectrum of model capacities, enabling smooth and elastic deployment across diverse memory budgets without the need for retraining. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_00942 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | SALAAD: Sparse And Low-Rank Adaptation via ADMM for Large Language Model Inference Ma, Hao Bal, Melis Ilayda Zhang, Liang Li, Bingcong He, Niao Zeilinger, Melanie Muehlebach, Michael Machine Learning Modern large language models are increasingly deployed under compute and memory constraints, making flexible control of model capacity a central challenge. While sparse and low-rank structures naturally trade off capacity and performance, existing approaches often rely on heuristic designs that ignore layer and matrix heterogeneity or require model-specific architectural modifications. We propose SALAAD, a plug-and-play framework applicable to different model architectures that induces sparse and low-rank structures during training. By formulating structured weight learning under an augmented Lagrangian framework and introducing an adaptive controller that dynamically balances the training loss and structural constraints, SALAAD preserves the stability of standard training dynamics while enabling explicit control over the evolution of effective model capacity during training. Experiments across model scales show that SALAAD substantially reduces memory consumption during deployment while achieving performance comparable to ad-hoc methods. Moreover, a single training run yields a continuous spectrum of model capacities, enabling smooth and elastic deployment across diverse memory budgets without the need for retraining. |
| title | SALAAD: Sparse And Low-Rank Adaptation via ADMM for Large Language Model Inference |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2602.00942 |