Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2603.15990 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911521021362176 |
|---|---|
| author | Han, Xiaolong Neri, Ferrante Jiang, Zijian Wu, Fang Ye, Yanfang Yin, Lu Wang, Zehong |
| author_facet | Han, Xiaolong Neri, Ferrante Jiang, Zijian Wu, Fang Ye, Yanfang Yin, Lu Wang, Zehong |
| contents | Each LoRA checkpoint compactly stores task-specific updates in low-rank weight matrices, offering an efficient way to adapt large language models to new tasks and domains. In principle, these weights already encode what the adapter does and how well it performs. In this paper, we ask whether this information can be read directly from the weights, without running the base model or accessing training data. A key obstacle is that a single LoRA update can be factorized in infinitely many ways. Without resolving this ambiguity, models trained on the factors may fit the particular factorization rather than the underlying update. To this end, we propose \methodfull, which maps each LoRA update to a provably canonical form via QR decomposition followed by SVD, so that all equivalent factorizations share the same representation. The resulting components are then tokenized and processed by a Transformer to produce a weight-space embedding. Across language and vision LoRA collections, W2T achieves strong results on attribute classification, performance prediction, and adapter retrieval, demonstrating that LoRA weights reliably indicate model behavior once factorization ambiguity is removed. Code is available at https://github.com/xiaolonghan2000/Weight2Token. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_15990 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | W2T: LoRA Weights Already Know What They Can Do Han, Xiaolong Neri, Ferrante Jiang, Zijian Wu, Fang Ye, Yanfang Yin, Lu Wang, Zehong Machine Learning Each LoRA checkpoint compactly stores task-specific updates in low-rank weight matrices, offering an efficient way to adapt large language models to new tasks and domains. In principle, these weights already encode what the adapter does and how well it performs. In this paper, we ask whether this information can be read directly from the weights, without running the base model or accessing training data. A key obstacle is that a single LoRA update can be factorized in infinitely many ways. Without resolving this ambiguity, models trained on the factors may fit the particular factorization rather than the underlying update. To this end, we propose \methodfull, which maps each LoRA update to a provably canonical form via QR decomposition followed by SVD, so that all equivalent factorizations share the same representation. The resulting components are then tokenized and processed by a Transformer to produce a weight-space embedding. Across language and vision LoRA collections, W2T achieves strong results on attribute classification, performance prediction, and adapter retrieval, demonstrating that LoRA weights reliably indicate model behavior once factorization ambiguity is removed. Code is available at https://github.com/xiaolonghan2000/Weight2Token. |
| title | W2T: LoRA Weights Already Know What They Can Do |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2603.15990 |