Context Distillation as Latent Memory Management
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916056505778176 |
|---|---|
| author | Zheng, Ziyang Li, Zeju Wen, Xiangyu Zhong, Jianyuan Huang, Junhua Chen, Lei Yuan, Mingxuan Xu, Qiang |
| author_facet | Zheng, Ziyang Li, Zeju Wen, Xiangyu Zhong, Jianyuan Huang, Junhua Chen, Lei Yuan, Mingxuan Xu, Qiang |
| contents | Context distillation compresses contextual information into model parameters, yet existing methods often ignore how multiple distilled latent memories should be stored, retrieved, and safely activated in non-oracle settings. We formulate context distillation as a latent memory management problem. We distill each context into an independent LoRA adapter, forming a modular memory bank that enables explicit memory selection. Given a query, our framework retrieves candidate memories, routes the query to the most suitable adapter, and uses a Self-Gating mechanism to decide whether latent memory should be activated. To improve efficiency, we further introduce cache sharing to reduce management overhead during inference. Experiments show that our method substantially outperforms baselines with retrieval, while Self-Gating improves robustness by deactivate unnecessary latent memories. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_28889 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Context Distillation as Latent Memory Management Zheng, Ziyang Li, Zeju Wen, Xiangyu Zhong, Jianyuan Huang, Junhua Chen, Lei Yuan, Mingxuan Xu, Qiang Machine Learning Artificial Intelligence Context distillation compresses contextual information into model parameters, yet existing methods often ignore how multiple distilled latent memories should be stored, retrieved, and safely activated in non-oracle settings. We formulate context distillation as a latent memory management problem. We distill each context into an independent LoRA adapter, forming a modular memory bank that enables explicit memory selection. Given a query, our framework retrieves candidate memories, routes the query to the most suitable adapter, and uses a Self-Gating mechanism to decide whether latent memory should be activated. To improve efficiency, we further introduce cache sharing to reduce management overhead during inference. Experiments show that our method substantially outperforms baselines with retrieval, while Self-Gating improves robustness by deactivate unnecessary latent memories. |
| title | Context Distillation as Latent Memory Management |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2605.28889 |