Cover Image

Optimal Decay Spectra for Linear Recurrences

Fuente: arXiv

Saved in:

Bibliographic Details
Main Author:	Cao, Yang
Format:	Preprint
Published:	2026
Subjects:	Machine Learning Artificial Intelligence Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Liger: Linearizing Large Language Models to Gated Recurrent Structures
by: Lan, Disen, et al.
Published: (2025)

AlphaDecay: Module-wise Weight Decay for Heavy-Tailed Balancing in LLMs
by: He, Di, et al.
Published: (2025)

Training Tensor Attention Efficiently: From Cubic to Almost Linear Time
by: Cao, Yang, et al.
Published: (2024)

Latent Chain-of-Thought? Decoding the Depth-Recurrent Transformer
by: Lu, Wenquan, et al.
Published: (2025)

Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMs
by: Bergsma, Shane, et al.
Published: (2025)

On the Robustness of Transformers against Context Hijacking for Linear Classification
by: Li, Tianle, et al.
Published: (2025)

RoPE Attention Can Be Trained in Almost Linear Time
by: Cao, Yang, et al.
Published: (2024)

Understanding Dynamic Compute Allocation in Recurrent Transformers
by: Moosa, Ibraheem Muhammad, et al.
Published: (2026)

Universal In-Context Approximation By Prompting Fully Recurrent Models
by: Petrov, Aleksandar, et al.
Published: (2024)

GateLoop: Fully Data-Controlled Linear Recurrence for Sequence Modeling
by: Katsch, Tobias
Published: (2023)

Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training
by: Bergsma, Shane, et al.
Published: (2025)

Thinking Deeper, Not Longer: Depth-Recurrent Transformers for Compositional Generalization
by: Chen, Hung-Hsuan
Published: (2026)

Loop, Think, & Generalize: Implicit Reasoning in Recurrent-Depth Transformers
by: Kohli, Harsh, et al.
Published: (2026)

Recurrent Knowledge Identification and Fusion for Language Model Continual Learning
by: Feng, Yujie, et al.
Published: (2025)

Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence
by: McLeish, Sean, et al.
Published: (2025)

How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining
by: Luo, Kairong, et al.
Published: (2025)

Depth-Recurrent Attention Mixtures: Giving Latent Reasoning the Attention it Deserves
by: Knupp, Jonas, et al.
Published: (2026)

Distil-xLSTM: Learning Attention Mechanisms through Recurrent Structures
by: Thiombiano, Abdoul Majid O., et al.
Published: (2025)

RecurrentGemma: Moving Past Transformers for Efficient Open Language Models
by: Botev, Aleksandar, et al.
Published: (2024)

xLSTM 7B: A Recurrent LLM for Fast and Efficient Inference
by: Beck, Maximilian, et al.
Published: (2025)

HoPE: A Novel Positional Encoding Without Long-Term Decay for Enhanced Context Awareness and Extrapolation
by: Chen, Yuhan, et al.
Published: (2024)

Higher-order Linear Attention
by: Zhang, Yifan, et al.
Published: (2025)

Your Transformer is Secretly Linear
by: Razzhigaev, Anton, et al.
Published: (2024)

RecMem: Recurrence-based Memory Consolidation for Efficient and Effective Long-Running LLM Agents
by: Dai, Zijie, et al.
Published: (2026)

CosmicFish-HRM: Adaptive Reasoning via Hierarchical Recurrent Mechanisms in Compact Language Models
by: Lakkapragada, Venkat Akhil
Published: (2026)

In Search of Needles in a 11M Haystack: Recurrent Memory Finds What LLMs Miss
by: Kuratov, Yuri, et al.
Published: (2024)

When Linear Attention Meets Autoregressive Decoding: Towards More Effective and Efficient Linearized Large Language Models
by: You, Haoran, et al.
Published: (2024)

PromptIntern: Saving Inference Costs by Internalizing Recurrent Prompt during Large Language Model Fine-tuning
by: Zou, Jiaru, et al.
Published: (2024)

Equivalent Linear Mappings of Large Language Models
by: Golden, James R.
Published: (2025)

Parallax: Parameterized Local Linear Attention for Language Modeling
by: Zuo, Yifei, et al.
Published: (2026)

The Markovian Thinker: Architecture-Agnostic Linear Scaling of Reasoning
by: Aghajohari, Milad, et al.
Published: (2025)

MoM: Linear Sequence Modeling with Mixture-of-Memories
by: Du, Jusen, et al.
Published: (2025)

Enhancing In-context Learning via Linear Probe Calibration
by: Abbas, Momin, et al.
Published: (2024)

The Linear Representation Hypothesis and the Geometry of Large Language Models
by: Park, Kiho, et al.
Published: (2023)

MesaNet: Sequence Modeling by Locally Optimal Test-Time Training
by: von Oswald, Johannes, et al.
Published: (2025)

Scaling Optimal LR Across Token Horizons
by: Bjorck, Johan, et al.
Published: (2024)

Training Optimal Large Diffusion Language Models
by: Ni, Jinjie, et al.
Published: (2025)

Rhetorical Questions in LLM Representations: A Linear Probing Study
by: Yao, Louie Hong, et al.
Published: (2026)

LoLCATs: On Low-Rank Linearizing of Large Language Models
by: Zhang, Michael, et al.
Published: (2024)

Cross-Architecture Transfer Learning for Linear-Cost Inference Transformers
by: Choi, Sehyun
Published: (2024)