Salvato in:
| Autori principali: | , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2604.21724 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910161786896384 |
|---|---|
| author | Chen, Yilong Xie, Yanxi Gao, Zitian Xin, He Xiao, Yihao Liu, Jason Klein Luo, Haoming Luo, Yifan Ye, Zhengmao Liu, Tingwen Zhao, Xin Tao, Ran Dai, Bryan |
| author_facet | Chen, Yilong Xie, Yanxi Gao, Zitian Xin, He Xiao, Yihao Liu, Jason Klein Luo, Haoming Luo, Yifan Ye, Zhengmao Liu, Tingwen Zhao, Xin Tao, Ran Dai, Bryan |
| contents | Large token-indexed lookup tables provide a compute-decoupled scaling path, but their practical gains are often limited by poor parameter efficiency and rapid memory growth. We attribute these limitations to Zipfian under-training of the long tail, heterogeneous demand across layers, and "slot collapse" that produces redundant embeddings. To address this, we propose X-GRAM, a frequency-aware dynamic token-injection framework. X-GRAM employs hybrid hashing and alias mixing to compress the tail while preserving head capacity, and refines retrieved vectors via normalized SwiGLU ShortConv to extract diverse local n-gram features. These signals are integrated into attention value streams and inter-layer residuals using depth-aware gating, effectively aligning static memory with dynamic context. This design introduces a memory-centric scaling axis that decouples model capacity from FLOPs. Extensive evaluations at the 0.73B and 1.15B scales show that X-GRAM improves average accuracy by as much as 4.4 points over the vanilla backbone and 3.2 points over strong retrieval baselines, while using substantially smaller tables in the 50% configuration. Overall, by decoupling capacity from compute through efficient memory management, X-GRAM offers a scalable and practical paradigm for future memory-augmented architectures. Code aviliable in https://github.com/Longyichen/X-gram. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_21724 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Beyond N-gram: Data-Aware X-GRAM Extraction for Efficient Embedding Parameter Scaling Chen, Yilong Xie, Yanxi Gao, Zitian Xin, He Xiao, Yihao Liu, Jason Klein Luo, Haoming Luo, Yifan Ye, Zhengmao Liu, Tingwen Zhao, Xin Tao, Ran Dai, Bryan Computation and Language Large token-indexed lookup tables provide a compute-decoupled scaling path, but their practical gains are often limited by poor parameter efficiency and rapid memory growth. We attribute these limitations to Zipfian under-training of the long tail, heterogeneous demand across layers, and "slot collapse" that produces redundant embeddings. To address this, we propose X-GRAM, a frequency-aware dynamic token-injection framework. X-GRAM employs hybrid hashing and alias mixing to compress the tail while preserving head capacity, and refines retrieved vectors via normalized SwiGLU ShortConv to extract diverse local n-gram features. These signals are integrated into attention value streams and inter-layer residuals using depth-aware gating, effectively aligning static memory with dynamic context. This design introduces a memory-centric scaling axis that decouples model capacity from FLOPs. Extensive evaluations at the 0.73B and 1.15B scales show that X-GRAM improves average accuracy by as much as 4.4 points over the vanilla backbone and 3.2 points over strong retrieval baselines, while using substantially smaller tables in the 50% configuration. Overall, by decoupling capacity from compute through efficient memory management, X-GRAM offers a scalable and practical paradigm for future memory-augmented architectures. Code aviliable in https://github.com/Longyichen/X-gram. |
| title | Beyond N-gram: Data-Aware X-GRAM Extraction for Efficient Embedding Parameter Scaling |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2604.21724 |