Image de couverture de livre

The Recurrent Transformer: Greater Effective Depth and Efficient Decoding

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Oncescu, Costin-Andrei, Morwani, Depen, Jelassi, Samy, Meterez, Alexandru, Kwun, Mujin, Kakade, Sham
Format:	Preprint
Publié:	2026
Sujets:	Machine Learning
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

A Simplified Analysis of SGD for Linear Regression with Weight Averaging
par: Meterez, Alexandru, et autres
Publié: (2025)

Seesaw: Accelerating Training by Balancing Learning Rate and Batch Size Scheduling
par: Meterez, Alexandru, et autres
Publié: (2025)

Feature emergence via margin maximization: case studies in algebraic tasks
par: Morwani, Depen, et autres
Publié: (2023)

Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averaging
par: Meterez, Alexandru, et autres
Publié: (2026)

LOTION: Smoothing the Optimization Landscape for Quantized Training
par: Kwun, Mujin, et autres
Publié: (2025)

Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
par: Zhao, Rosie, et autres
Publié: (2025)

SOAP: Improving and Stabilizing Shampoo using Adam
par: Vyas, Nikhil, et autres
Publié: (2024)

Connections between Schedule-Free Optimizers, AdEMAMix, and Accelerated SGD Variants
par: Morwani, Depen, et autres
Publié: (2025)

The Potential of Second-Order Optimization for LLMs: A Study with Full Gauss-Newton
par: Abreu, Natalie, et autres
Publié: (2025)

Flash Inference: Near Linear Time Inference for Long Convolution Sequence Models and Beyond
par: Oncescu, Costin-Andrei, et autres
Publié: (2024)

Matching Features, Not Tokens: Energy-Based Fine-Tuning of Language Models
par: Jelassi, Samy, et autres
Publié: (2026)

Deconstructing What Makes a Good Optimizer for Language Models
par: Zhao, Rosie, et autres
Publié: (2024)

Repeat After Me: Transformers are Better than State Space Models at Copying
par: Jelassi, Samy, et autres
Publié: (2024)

Beyond Implicit Bias: The Insignificance of SGD Noise in Online Learning
par: Vyas, Nikhil, et autres
Publié: (2023)

Characterization and Mitigation of Training Instabilities in Microscaling Formats
par: Su, Huangyuan, et autres
Publié: (2025)

A New Perspective on Shampoo's Preconditioner
par: Morwani, Depen, et autres
Publié: (2024)

The Role of Sparsity for Length Generalization in Transformers
par: Golowich, Noah, et autres
Publié: (2025)

Universal Length Generalization with Turing Programs
par: Hou, Kaiying, et autres
Publié: (2024)

Adam or Gauss-Newton? A Comparative Study In Terms of Basis Alignment and SGD Noise
par: Liu, Bingbin, et autres
Publié: (2025)

Q-Probe: A Lightweight Approach to Reward Maximization for Language Models
par: Li, Kenneth, et autres
Publié: (2024)

LoRA Soups: Merging LoRAs for Practical Skill Composition Tasks
par: Prabhakar, Akshara, et autres
Publié: (2024)

How Does Critical Batch Size Scale in Pre-training?
par: Zhang, Hanlin, et autres
Publié: (2024)

Opportunistic Expert Activation: Batch-Aware Expert Routing for Faster Decode Without Retraining
par: Oncescu, Costin-Andrei, et autres
Publié: (2025)

Collective Model Intelligence Requires Compatible Specialization
par: Pari, Jyothish, et autres
Publié: (2024)

Mixture of Parrots: Experts improve memorization more than reasoning
par: Jelassi, Samy, et autres
Publié: (2024)

How Does Overparameterization Affect Features?
par: Duzgun, Ahmet Cagri, et autres
Publié: (2024)

Super Consistency of Neural Network Landscapes and Learning Rate Transfer
par: Noci, Lorenzo, et autres
Publié: (2024)

The Optimization Landscape of SGD Across the Feature Learning Strength
par: Atanasov, Alexander, et autres
Publié: (2024)

To Backtrack or Not to Backtrack: When Sequential Search Limits Model Reasoning
par: Qin, Tian, et autres
Publié: (2025)

Latent Chain-of-Thought? Decoding the Depth-Recurrent Transformer
par: Lu, Wenquan, et autres
Publié: (2025)

GQ-VAE: A gated quantized VAE for learning variable length tokens
par: Datta, Theo, et autres
Publié: (2025)

Koopman Spectrum Nonlinear Regulators and Efficient Online Learning
par: Ohnishi, Motoya, et autres
Publié: (2021)

Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusions
par: Kim, Jaeyeon, et autres
Publié: (2025)

Prescriptive Scaling Reveals the Evolution of Language Model Capabilities
par: Zhang, Hanlin, et autres
Publié: (2026)

Discovering Hierarchical Latent Capabilities of Language Models via Causal Representation Learning
par: Jin, Jikai, et autres
Publié: (2025)

Learning Hidden Markov Models Using Conditional Samples
par: Kakade, Sham M., et autres
Publié: (2023)

Matching the Statistical Query Lower Bound for $k$-Sparse Parity Problems with Sign Stochastic Gradient Descent
par: Kou, Yiwen, et autres
Publié: (2024)

Causal discovery for linear causal model with correlated noise: an Adversarial Learning Approach
par: Zhou, Mujin, et autres
Publié: (2026)

Stop Training for the Worst: Progressive Unmasking Accelerates Masked Diffusion Training
par: Kim, Jaeyeon, et autres
Publié: (2026)

In Good GRACEs: Principled Teacher Selection for Knowledge Distillation
par: Panigrahi, Abhishek, et autres
Publié: (2025)