Imagen de Portada

RAM-Net: Expressive Linear Attention with Selectively Addressable Memory

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Xiao, Kaicheng, Li, Haotian, Dong, Liran, Xing, Guoliang
Formato:	Preprint
Publicado:	2026
Materias:	Machine Learning Computation and Language
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Kimi Linear: An Expressive, Efficient Attention Architecture
por: Kimi Team, et al.
Publicado: (2025)

The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicry
por: Zhang, Michael, et al.
Publicado: (2024)

Stochastic Attention: Connectome-Inspired Randomized Routing for Expressive Linear-Time Attention
por: Jin, Zehao, et al.
Publicado: (2026)

Taipan: Efficient and Expressive State Space Language Models with Selective Attention
por: Van Nguyen, Chien, et al.
Publicado: (2024)

Linear Attention Sequence Parallelism
por: Sun, Weigao, et al.
Publicado: (2024)

More Expressive Attention with Negative Weights
por: Lv, Ang, et al.
Publicado: (2024)

LoMA: Lossless Compressed Memory Attention
por: Wang, Yumeng, et al.
Publicado: (2024)

Why Softmax Attention Outperforms Linear Attention
por: Deng, Yichuan, et al.
Publicado: (2023)

Scaling Linear Attention with Sparse State Expansion
por: Pan, Yuqi, et al.
Publicado: (2025)

Don't Read Everything: A Curvature-Conditioned Query for Linear Attention
por: Le, Dong, et al.
Publicado: (2026)

Learning a Continue-Thinking Token for Enhanced Test-Time Scaling
por: Ringel, Liran, et al.
Publicado: (2025)

SEA: Sparse Linear Attention with Estimated Attention Mask
por: Lee, Heejun, et al.
Publicado: (2023)

Theoretical Constraints on the Expressive Power of $\mathsf{RoPE}$-based Tensor Attention Transformers
por: Li, Xiaoyu, et al.
Publicado: (2024)

S$^3$-Attention:Attention-Aligned Endogenous Retrieval for Memory-Bounded Long-Context Inference
por: Ma, Qingsen, et al.
Publicado: (2026)

OSDN: Improving Delta Rule with Provable Online Preconditioning in Linear Attention
por: Zhou, Chenyu, et al.
Publicado: (2026)

Neural Attention Search Linear: Towards Adaptive Token-Level Hybrid Attention Models
por: Deng, Difan, et al.
Publicado: (2026)

Alleviating Forgetfulness of Linear Attention by Hybrid Sparse Attention and Contextualized Learnable Token Eviction
por: He, Mutian, et al.
Publicado: (2025)

On the Expressiveness and Length Generalization of Selective State-Space Models on Regular Languages
por: Terzić, Aleksandar, et al.
Publicado: (2024)

Gated Linear Attention Transformers with Hardware-Efficient Training
por: Yang, Songlin, et al.
Publicado: (2023)

Trading Complexity for Expressivity Through Structured Generalized Linear Token Mixing
por: Fagnou, Erwan, et al.
Publicado: (2026)

Selective Attention: Enhancing Transformer through Principled Context Control
por: Zhang, Xuechen, et al.
Publicado: (2024)

Wave-PDE Nets: Trainable Wave-Equation Layers as an Alternative to Attention
por: Vejendla, Harshil
Publicado: (2025)

BitNet Distillation
por: Wu, Xun, et al.
Publicado: (2025)

Trellis: Learning to Compress Key-Value Memory in Attention Models
por: Karami, Mahdi, et al.
Publicado: (2025)

Unifying Linear-Time Attention via Latent Probabilistic Modelling
por: Dolga, Rares, et al.
Publicado: (2024)

Transformer Based Linear Attention with Optimized GPU Kernel Implementation
por: Gerami, Armin, et al.
Publicado: (2025)

LoLA: Low-Rank Linear Attention With Sparse Caching
por: McDermott, Luke, et al.
Publicado: (2025)

Efficient RLVR Training via Weighted Mutual Information Data Selection
por: Zhou, Xinyu, et al.
Publicado: (2026)

Ladder Up, Memory Down: Low-Cost Fine-Tuning With Side Nets
por: Zheng, Estelle, et al.
Publicado: (2025)

PowerAttention: Exponentially Scaling of Receptive Fields for Effective Sparse Attention
por: Chen, Lida, et al.
Publicado: (2025)

Higher-order Linear Attention
por: Zhang, Yifan, et al.
Publicado: (2025)

Attention with Trained Embeddings Provably Selects Important Tokens
por: Wu, Diyuan, et al.
Publicado: (2025)

Learning to Focus: Focal Attention for Selective and Scalable Transformers
por: Ram, Dhananjay, et al.
Publicado: (2025)

Improving Transformers with Dynamically Composable Multi-Head Attention
por: Xiao, Da, et al.
Publicado: (2024)

Memory-Efficient Fine-Tuning of Transformers via Token Selection
por: Simoulin, Antoine, et al.
Publicado: (2025)

Immunogenicity Prediction with Dual Attention Enables Vaccine Target Selection
por: Li, Song, et al.
Publicado: (2024)

Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models
por: De, Soham, et al.
Publicado: (2024)

Gated KalmaNet: A Fading Memory Layer Through Test-Time Ridge Regression
por: Peng, Liangzu, et al.
Publicado: (2025)

Generalized Probabilistic Attention Mechanism in Transformers
por: Heo, DongNyeong, et al.
Publicado: (2024)

Post-LayerNorm Is Back: Stable, ExpressivE, and Deep
por: Chen, Chen, et al.
Publicado: (2026)