Imagen de Portada

Delayed Attention Training Improves Length Generalization in Transformer--RNN Hybrids

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Phan, Buu, Ebrahimi, Reza, Haresh, Sanjay, Memisevic, Roland
Formato:	Preprint
Publicado:	2025
Materias:	Machine Learning
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

ClevrSkills: Compositional Language and Visual Reasoning in Robotics
por: Haresh, Sanjay, et al.
Publicado: (2024)

Revisiting Bi-Linear State Transitions in Recurrent Neural Networks
por: Ebrahimi, M. Reza, et al.
Publicado: (2025)

On the "Induction Bias" in Sequence Models
por: Ebrahimi, M. Reza, et al.
Publicado: (2026)

Channel Simulation and Distributed Compression with Ensemble Rejection Sampling
por: Phan, Buu, et al.
Publicado: (2025)

List-Level Distribution Coupling with Applications to Speculative Decoding and Lossy Compression
por: Rowan, Joseph, et al.
Publicado: (2025)

Attention as an RNN
por: Feng, Leo, et al.
Publicado: (2024)

Notes-to-Self: Scratchpad Augmented VLAs for Memory Dependent Manipulation Tasks
por: Haresh, Sanjay, et al.
Publicado: (2026)

Cross-Tokenizer Likelihood Scoring Algorithms for Language Model Distillation
por: Phan, Buu, et al.
Publicado: (2025)

Multi-Draft Speculative Sampling: Canonical Decomposition and Theoretical Limits
por: Khisti, Ashish, et al.
Publicado: (2024)

Your Context Is Not an Array: Unveiling Random Access Limitations in Transformers
por: Ebrahimi, MohammadReza, et al.
Publicado: (2024)

Replacing thinking with tool usage enables reasoning in small language models
por: Rainone, Corrado, et al.
Publicado: (2025)

Understanding and Mitigating Tokenization Bias in Language Models
por: Phan, Buu, et al.
Publicado: (2024)

Aligning Robot Navigation Behaviors with Human Intentions and Preferences
por: Karnan, Haresh
Publicado: (2024)

Look, Remember and Reason: Grounded reasoning in videos with language models
por: Bhattacharyya, Apratim, et al.
Publicado: (2023)

Enhancing Time Series Forecasting with Fuzzy Attention-Integrated Transformers
por: Chakraborty, Sanjay, et al.
Publicado: (2025)

Attention Augmented GNN RNN-Attention Models for Advanced Cybersecurity Intrusion Detection
por: Biradar, Jayant, et al.
Publicado: (2025)

Understanding and Improving Length Generalization in Hierarchical Sparse Attention Models
por: Leng, Jiaqi, et al.
Publicado: (2025)

TRA: Better Length Generalisation with Threshold Relative Attention
por: Opper, Mattia, et al.
Publicado: (2025)

Looped Transformers for Length Generalization
por: Fan, Ying, et al.
Publicado: (2024)

Learning to Forget with Information Divergence Reweighted Objectives for Noisy Labels
por: Birrell, Jeremiah, et al.
Publicado: (2025)

Optimal Transport Regularized Divergences: Application to Adversarial Robustness
por: Birrell, Jeremiah, et al.
Publicado: (2023)

Integrating Quantum-Classical Attention in Patch Transformers for Enhanced Time Series Forecasting
por: Chakraborty, Sanjay, et al.
Publicado: (2025)

Exact Byte-Level Probabilities from Tokenized Language Models for FIM-Tasks and Model Ensembles
por: Phan, Buu, et al.
Publicado: (2024)

Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challenges
por: Lee, Nayoung, et al.
Publicado: (2025)

Quantitative Bounds for Length Generalization in Transformers
por: Izzo, Zachary, et al.
Publicado: (2025)

When Learning Hurts: Fixed-Pole RNN for Real-Time Online Training
por: Morgan, Alexander, et al.
Publicado: (2026)

Understanding and Improving Length Generalization in Recurrent Models
por: Ruiz, Ricardo Buitrago, et al.
Publicado: (2025)

The Norm-Separation Delay Law of Grokking: A First-Principles Theory of Delayed Generalization
por: Khanh, Truong Xuan, et al.
Publicado: (2026)

Crash Severity Prediction Using Deep Learning Approaches: A Hybrid CNN-RNN Framework
por: Koohfar, Sahar
Publicado: (2025)

On Vanishing Variance in Transformer Length Generalization
por: Li, Ruining, et al.
Publicado: (2025)

Tiled Flash Linear Attention: More Efficient Linear RNN and xLSTM Kernels
por: Beck, Maximilian, et al.
Publicado: (2025)

A Formal Framework for Understanding Length Generalization in Transformers
por: Huang, Xinting, et al.
Publicado: (2024)

Arithmetic Transformers Can Length-Generalize in Both Operand Length and Count
por: Cho, Hanseul, et al.
Publicado: (2024)

Focusing on What Matters: Object-Agent-centric Tokenization for Vision Language Action models
por: Bendikas, Rokas, et al.
Publicado: (2025)

Leakage and Second-Order Dynamics Improve Hippocampal RNN Replay
por: Casco-Rodriguez, Josue, et al.
Publicado: (2026)

Spectral Conditioning of Attention Improves Transformer Performance
por: Saratchandran, Hemanth, et al.
Publicado: (2026)

HyMaTE: A Hybrid Mamba and Transformer Model for EHR Representation Learning
por: Mottalib, Md Mozaharul, et al.
Publicado: (2025)

Position Coupling: Improving Length Generalization of Arithmetic Transformers Using Task Structure
por: Cho, Hanseul, et al.
Publicado: (2024)

On Self-Adaptive Perception Loss Function for Sequential Lossy Compression
por: Salehkalaibar, Sadaf, et al.
Publicado: (2025)

Improving Variable-Length Generation in Diffusion Language Models via Length Regularization
por: Cheng, Zicong, et al.
Publicado: (2026)