SHINE: A Scalable In-Context Hypernetwork for Mapping Context to LoRA in a Single Pass
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917516161318912 |
|---|---|
| author | Liu, Yewei Wang, Xiyuan Mao, Yansheng Gelbery, Yoav Maron, Haggai Zhang, Muhan |
| author_facet | Liu, Yewei Wang, Xiyuan Mao, Yansheng Gelbery, Yoav Maron, Haggai Zhang, Muhan |
| contents | We propose SHINE (Scalable Hyper In-context NEtwork), a scalable hypernetwork that can map diverse meaningful contexts into high-quality LoRA adapters for large language models (LLMs). By reusing the frozen LLM's own parameters in an in-context hypernetwork design and introducing architectural innovations, SHINE overcomes key limitations of prior hypernetworks and achieves strong expressive power with a relatively small number of parameters. We introduce a pretraining and instruction fine-tuning pipeline, and train our hypernetwork to generate high quality LoRA adapters from diverse meaningful contexts in a single forward pass. It updates LLM parameters without any fine-tuning, and immediately enables complex question answering tasks related to the context without directly accessing the context, effectively transforming in-context knowledge to in-parameter knowledge in one pass. Our work achieves outstanding results on various tasks, greatly saves time, computation and memory costs compared to SFT-based LLM adaptation, and shows great potential for scaling. Our code is available at https://github.com/MuLabPKU/SHINE |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_06358 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | SHINE: A Scalable In-Context Hypernetwork for Mapping Context to LoRA in a Single Pass Liu, Yewei Wang, Xiyuan Mao, Yansheng Gelbery, Yoav Maron, Haggai Zhang, Muhan Computation and Language Artificial Intelligence We propose SHINE (Scalable Hyper In-context NEtwork), a scalable hypernetwork that can map diverse meaningful contexts into high-quality LoRA adapters for large language models (LLMs). By reusing the frozen LLM's own parameters in an in-context hypernetwork design and introducing architectural innovations, SHINE overcomes key limitations of prior hypernetworks and achieves strong expressive power with a relatively small number of parameters. We introduce a pretraining and instruction fine-tuning pipeline, and train our hypernetwork to generate high quality LoRA adapters from diverse meaningful contexts in a single forward pass. It updates LLM parameters without any fine-tuning, and immediately enables complex question answering tasks related to the context without directly accessing the context, effectively transforming in-context knowledge to in-parameter knowledge in one pass. Our work achieves outstanding results on various tasks, greatly saves time, computation and memory costs compared to SFT-based LLM adaptation, and shows great potential for scaling. Our code is available at https://github.com/MuLabPKU/SHINE |
| title | SHINE: A Scalable In-Context Hypernetwork for Mapping Context to LoRA in a Single Pass |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2602.06358 |