Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.08798 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914085735497728 |
|---|---|
| author | Rafiuddin, S M Khan, Muntaha Nujat |
| author_facet | Rafiuddin, S M Khan, Muntaha Nujat |
| contents | Transformer attention scales quadratically with sequence length O(n^2), limiting long-context use. We propose Adaptive Retention, a probabilistic, layer-wise token selection mechanism that learns which representations to keep under a strict global budget M. Retention is modeled with Bernoulli gates trained via a Hard-Concrete/variational relaxation and enforced with a simple top-M rule at inference, making the method differentiable and drop-in for standard encoders. Across classification, extractive QA, and long-document summarization, keeping only 30-50% of tokens preserves >= 95% of full-model performance while cutting peak memory by ~35-45% and improving throughput by up to ~1.8x. This architecture-agnostic approach delivers practical long-context efficiency without modifying base attention or task heads. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_08798 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Learning What to Remember: Adaptive Probabilistic Memory Retention for Memory-Efficient Language Models Rafiuddin, S M Khan, Muntaha Nujat Computation and Language Transformer attention scales quadratically with sequence length O(n^2), limiting long-context use. We propose Adaptive Retention, a probabilistic, layer-wise token selection mechanism that learns which representations to keep under a strict global budget M. Retention is modeled with Bernoulli gates trained via a Hard-Concrete/variational relaxation and enforced with a simple top-M rule at inference, making the method differentiable and drop-in for standard encoders. Across classification, extractive QA, and long-document summarization, keeping only 30-50% of tokens preserves >= 95% of full-model performance while cutting peak memory by ~35-45% and improving throughput by up to ~1.8x. This architecture-agnostic approach delivers practical long-context efficiency without modifying base attention or task heads. |
| title | Learning What to Remember: Adaptive Probabilistic Memory Retention for Memory-Efficient Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2510.08798 |