Imagen de Portada

A Formal Framework for Understanding Length Generalization in Transformers

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Huang, Xinting, Yang, Andy, Bhattamishra, Satwik, Sarrof, Yash, Krebs, Andreas, Zhou, Hattie, Nakkiran, Preetum, Hahn, Michael
Formato:	Preprint
Publicado:	2024
Materias:	Machine Learning
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Discovering Interpretable Algorithms by Decompiling Transformers to RASP
por: Huang, Xinting, et al.
Publicado: (2026)

Step-by-Step Diffusion: An Elementary Tutorial
por: Nakkiran, Preetum, et al.
Publicado: (2024)

Separations in the Representational Capabilities of Transformers and Recurrent Architectures
por: Bhattamishra, Satwik, et al.
Publicado: (2024)

The Expressive Capacity of State Space Models: A Formal Language Perspective
por: Sarrof, Yash, et al.
Publicado: (2024)

Provably Learning Attention with Queries
por: Bhattamishra, Satwik, et al.
Publicado: (2026)

Classifier-Free Guidance is a Predictor-Corrector
por: Bradley, Arwen, et al.
Publicado: (2024)

Barriers to Universal Reasoning With Transformers (And How to Overcome Them)
por: Kraus, Oliver, et al.
Publicado: (2026)

Hardness of Learning Regular Languages in the Next Symbol Prediction Setting
por: Bhattamishra, Satwik, et al.
Publicado: (2025)

Vanishing Gradients in Reinforcement Finetuning of Language Models
por: Razin, Noam, et al.
Publicado: (2023)

When is Multicalibration Post-Processing Necessary?
por: Hansen, Dutch, et al.
Publicado: (2024)

On the Ability of Transformers to Verify Plans
por: Sarrof, Yash, et al.
Publicado: (2026)

Born a Transformer -- Always a Transformer? On the Effect of Pretraining on Architectural Abilities
por: Jobanputra, Mayank, et al.
Publicado: (2025)

Benefits and Limitations of Communication in Multi-Agent Reasoning
por: Rizvi-Martel, Michael, et al.
Publicado: (2025)

The Transformer Cookbook
por: Yang, Andy, et al.
Publicado: (2025)

Mechanisms of Projective Composition of Diffusion Models
por: Bradley, Arwen, et al.
Publicado: (2025)

Trained on Tokens, Calibrated on Concepts: The Emergence of Semantic Calibration in LLMs
por: Nakkiran, Preetum, et al.
Publicado: (2025)

Composition and Control with Distilled Energy Diffusion Models and Sequential Monte Carlo
por: Thornton, James, et al.
Publicado: (2025)

Decomposing Representation Space into Interpretable Subspaces with Unsupervised Learning
por: Huang, Xinting, et al.
Publicado: (2025)

How JEPA Avoids Noisy Features: The Implicit Bias of Deep Linear Self Distillation Networks
por: Littwin, Etai, et al.
Publicado: (2024)

Lower Bounds for Chain-of-Thought Reasoning in Hard-Attention Transformers
por: Amiri, Alireza, et al.
Publicado: (2025)

InversionView: A General-Purpose Method for Reading Information from Neural Activations
por: Huang, Xinting, et al.
Publicado: (2024)

Benign, Tempered, or Catastrophic: A Taxonomy of Overfitting
por: Mallinar, Neil, et al.
Publicado: (2022)

Inshrinkerator: Compressing Deep Learning Training Checkpoints via Dynamic Quantization
por: Agrawal, Amey, et al.
Publicado: (2023)

Contextualize-then-Aggregate: Circuits for In-Context Learning in Gemma-2 2B
por: Bakalova, Aleksandra, et al.
Publicado: (2025)

Length Generalization Bounds for Transformers
por: Yang, Andy, et al.
Publicado: (2026)

Normalizing Flows are Capable Generative Models
por: Zhai, Shuangfei, et al.
Publicado: (2024)

On Vanishing Variance in Transformer Length Generalization
por: Li, Ruining, et al.
Publicado: (2025)

Looped Transformers for Length Generalization
por: Fan, Ying, et al.
Publicado: (2024)

MALT Diffusion: Memory-Augmented Latent Transformers for Any-Length Video Generation
por: Yu, Sihyun, et al.
Publicado: (2025)

Quantitative Bounds for Length Generalization in Transformers
por: Izzo, Zachary, et al.
Publicado: (2025)

Gating Enables Curvature: A Geometric Expressivity Gap in Attention
por: Bathula, Satwik, et al.
Publicado: (2026)

Understanding and Improving Length Generalization in Recurrent Models
por: Ruiz, Ricardo Buitrago, et al.
Publicado: (2025)

Why are Sensitive Functions Hard for Transformers?
por: Hahn, Michael, et al.
Publicado: (2024)

Trace Length is a Simple Uncertainty Signal in Reasoning Models
por: Devic, Siddartha, et al.
Publicado: (2025)

Transformers Can Achieve Length Generalization But Not Robustly
por: Zhou, Yongchao, et al.
Publicado: (2024)

STIQ: Safeguarding Training and Inferencing of Quantum Neural Networks from Untrusted Cloud
por: Kundu, Satwik, et al.
Publicado: (2024)

Inverse-Transpilation: Reverse-Engineering Quantum Compiler Optimization Passes from Circuit Snapshots
por: Kundu, Satwik, et al.
Publicado: (2025)

Arithmetic Transformers Can Length-Generalize in Both Operand Length and Count
por: Cho, Hanseul, et al.
Publicado: (2024)

Arbitrary-Length Generalization for Addition in a Tiny Transformer
por: Patriota, Alexandre Galvao
Publicado: (2024)

Security Concerns in Quantum Machine Learning as a Service
por: Kundu, Satwik, et al.
Publicado: (2024)