Imagen de Portada

One-layer transformers fail to solve the induction heads task

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Sanford, Clayton, Hsu, Daniel, Telgarsky, Matus
Formato:	Preprint
Publicado:	2024
Materias:	Machine Learning
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Transformers, parallel computation, and logarithmic depth
por: Sanford, Clayton, et al.
Publicado: (2024)

On Achieving Optimal Adversarial Test Error
por: Li, Justin D., et al.
Publicado: (2023)

Astral Space: Convex Analysis at Infinity
por: Dudík, Miroslav, et al.
Publicado: (2022)

Benefits of Early Stopping in Gradient Descent for Overparameterized Logistic Regression
por: Wu, Jingfeng, et al.
Publicado: (2025)

Spectrum Extraction and Clipping for Implicitly Linear Layers
por: Boroojeny, Ali Ebrahimpour, et al.
Publicado: (2024)

Large Stepsize Gradient Descent for Logistic Loss: Non-Monotonicity of the Loss Improves Optimization Efficiency
por: Wu, Jingfeng, et al.
Publicado: (2024)

Lower bounds for one-layer transformers that compute parity
por: Hsu, Daniel
Publicado: (2026)

Fast attention mechanisms: a tale of parallelism
por: Liu, Jingwen, et al.
Publicado: (2025)

Basic Inequalities for First-Order Optimization with Applications to Statistical Risk Analysis
por: Paik, Seunghoon, et al.
Publicado: (2025)

Stream separation improves Bregman conditioning in transformers
por: Kerce, James Clayton
Publicado: (2026)

When Do Transformers Outperform Feedforward and Recurrent Networks? A Statistical Perspective
por: Mousavi-Hosseini, Alireza, et al.
Publicado: (2025)

Attention layers provably solve single-location regression
por: Marion, Pierre, et al.
Publicado: (2024)

Lost in Tokenization: Fundamental Trade-offs in Graph Tokenization for Transformers
por: Bechler-Speicher, Maya, et al.
Publicado: (2026)

Fine-tuned network relies on generic representation to solve unseen cognitive task
por: Lin, Dongyan
Publicado: (2024)

Small transformer architectures for task switching
por: Gros, Claudius
Publicado: (2025)

What needs to go right for an induction head? A mechanistic study of in-context learning circuits and their formation
por: Singh, Aaditya K., et al.
Publicado: (2024)

Relational reasoning and inductive bias in transformers and large language models
por: Geerts, Jesse, et al.
Publicado: (2025)

A One-Inclusion Graph Approach to Multi-Group Learning
por: Bergam, Noah, et al.
Publicado: (2026)

Next-Token Prediction and Regret Minimization
por: Mohri, Mehryar, et al.
Publicado: (2026)

Best of Both Worlds: Advantages of Hybrid Graph Sequence Models
por: Behrouz, Ali, et al.
Publicado: (2024)

Provably learning a multi-head attention layer
por: Chen, Sitan, et al.
Publicado: (2024)

Out-of-distribution generalization via composition: a lens through induction heads in Transformers
por: Song, Jiajun, et al.
Publicado: (2024)

Tabular data generation with tensor contraction layers and transformers
por: Silva, Aníbal, et al.
Publicado: (2024)

Dynamic layer selection in decoder-only transformers
por: Glavas, Theodore, et al.
Publicado: (2024)

Understanding Transformer Reasoning Capabilities via Graph Algorithms
por: Sanford, Clayton, et al.
Publicado: (2024)

Depth-Width tradeoffs in Algorithmic Reasoning of Graph Tasks with Transformers
por: Yehudai, Gilad, et al.
Publicado: (2025)

Trapped by simplicity: When Transformers fail to learn from noisy features
por: Peters, Evan, et al.
Publicado: (2026)

Adversarial Debiasing for Unbiased Parameter Recovery
por: Sanford, Luke C, et al.
Publicado: (2025)

Learning on Transformers is Provable Low-Rank and Sparse: A One-layer Analysis
por: Li, Hongkang, et al.
Publicado: (2024)

From drift to adaptation to the failed ml model: Transfer Learning in Industrial MLOps
por: Ashraf, Waqar Muhammad, et al.
Publicado: (2026)

Propagation of Chaos in One-hidden-layer Neural Networks beyond Logarithmic Time
por: Glasgow, Margalit, et al.
Publicado: (2025)

Unified CNNs and transformers underlying learning mechanism reveals multi-head attention modus vivendi
por: Koresh, Ella, et al.
Publicado: (2025)

Shared-unique Features and Task-aware Prioritized Sampling on Multi-task Reinforcement Learning
por: Lin, Po-Shao, et al.
Publicado: (2024)

Imitation of human motion achieves natural head movements for humanoid robots in an active-speaker detection task
por: Ding, Bosong, et al.
Publicado: (2024)

Dimension lower bounds for linear approaches to function approximation
por: Hsu, Daniel
Publicado: (2025)

The Implicit Bias of Gradient Descent on Separable Multiclass Data
por: Ravi, Hrithik, et al.
Publicado: (2024)

Improving One-class Recommendation with Multi-tasking on Various Preference Intensities
por: Shao, Chu-Jen, et al.
Publicado: (2024)

Enhancing Motion Variation in Text-to-Motion Models via Pose and Video Conditioned Editing
por: Leite, Clayton, et al.
Publicado: (2024)

Flow Straight and Fast in Hilbert Space: Functional Rectified Flow
por: Zhang, Jianxin, et al.
Publicado: (2025)

Label Embedding via Low-Coherence Matrices
por: Zhang, Jianxin, et al.
Publicado: (2023)