Imagen de Portada

How Transformers Utilize Multi-Head Attention in In-Context Learning? A Case Study on Sparse Linear Regression

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Chen, Xingwu, Zhao, Lei, Zou, Difan
Formato:	Preprint
Publicado:	2024
Materias:	Machine Learning
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

What Can Transformer Learn with Varying Depth? Case Studies on Sequence Learning Tasks
por: Chen, Xingwu, et al.
Publicado: (2024)

Towards Theoretical Understanding of Transformer Test-Time Computing: Investigation on In-Context Linear Regression
por: Chen, Xingwu, et al.
Publicado: (2025)

On the Robustness of Transformers against Context Hijacking for Linear Classification
por: Li, Tianle, et al.
Publicado: (2025)

How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?
por: Wu, Jingfeng, et al.
Publicado: (2023)

Superiority of Multi-Head Attention in In-Context Linear Regression
por: Cui, Yingqian, et al.
Publicado: (2024)

Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection
por: Chen, Xingwu, et al.
Publicado: (2025)

Learning under Quantization for High-Dimensional Linear Regression
por: Zhang, Dechen, et al.
Publicado: (2025)

On the Interpolation Error of Nonlinear Attention versus Linear Regression
por: Liao, Zhenyu, et al.
Publicado: (2025)

In-Context Linear Regression Demystified: Training Dynamics and Mechanistic Interpretability of Multi-Head Softmax Attention
por: He, Jianliang, et al.
Publicado: (2025)

An In-depth Investigation of Sparse Rate Reduction in Transformer-like Models
por: Hu, Yunzhe, et al.
Publicado: (2024)

Scaling Laws for Precision in High-Dimensional Linear Regression
por: Zhang, Dechen, et al.
Publicado: (2026)

In-Context Learning in Linear vs. Quadratic Attention Models: An Empirical Study on Regression Tasks
por: Goel, Ayush, et al.
Publicado: (2026)

Exact Conversion of In-Context Learning to Model Weights in Linearized-Attention Transformers
por: Chen, Brian K, et al.
Publicado: (2024)

Beyond Linear Attention: Softmax Transformers Implement In-Context Reinforcement Learning
por: Xie, Zixuan, et al.
Publicado: (2026)

A Mechanism Study of Delayed Loss Spikes in Batch-Normalized Linear Models
por: Gao, Peifeng, et al.
Publicado: (2026)

Understanding In-Context Learning for Nonlinear Regression with Transformers: Attention as Featurizer
por: Hsu, Alexander, et al.
Publicado: (2026)

Dynamics of Transient Structure in In-Context Linear Regression Transformers
por: Carroll, Liam, et al.
Publicado: (2025)

The Atlas of In-Context Learning: How Attention Heads Shape In-Context Retrieval Augmentation
por: Kahardipraja, Patrick, et al.
Publicado: (2025)

InAttention: Linear Context Scaling for Transformers
por: Eisner, Joseph
Publicado: (2024)

Hyper-SET: Designing Transformers via Hyperspherical Energy Minimization
por: Hu, Yunzhe, et al.
Publicado: (2025)

Memorization Capacity of Multi-Head Attention in Transformers
por: Mahdavi, Sadegh, et al.
Publicado: (2023)

A Human-Like Reasoning Framework for Multi-Phases Planning Task with Large Language Models
por: Xie, Chengxing, et al.
Publicado: (2024)

Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regression
por: Fu, Deqing, et al.
Publicado: (2023)

Kernel Regression in Structured Non-IID Settings: Theory and Implications for Denoising Score Learning
por: Zhang, Dechen, et al.
Publicado: (2025)

Neural Attention Search Linear: Towards Adaptive Token-Level Hybrid Attention Models
por: Deng, Difan, et al.
Publicado: (2026)

Sparse Attention as Compact Kernel Regression
por: Santos, Saul, et al.
Publicado: (2026)

In-Context Compositional Learning via Sparse Coding Transformer
por: Chen, Wei, et al.
Publicado: (2025)

Sparse Linear Regression and Lattice Problems
por: Gupte, Aparna, et al.
Publicado: (2024)

On the Limitation and Experience Replay for GNNs in Continual Learning
por: Su, Junwei, et al.
Publicado: (2023)

Transformers Handle Endogeneity in In-Context Linear Regression
por: Liang, Haodong, et al.
Publicado: (2024)

Orion-MSP: Multi-Scale Sparse Attention for Tabular In-Context Learning
por: Bouadi, Mohamed, et al.
Publicado: (2025)

Improving Group Robustness on Spurious Correlation Requires Preciser Group Inference
por: Han, Yujin, et al.
Publicado: (2024)

Provable In-Context Learning of Nonlinear Regression with Transformers
por: Li, Hongbo, et al.
Publicado: (2025)

Training Dynamics of In-Context Learning in Linear Attention
por: Zhang, Yedi, et al.
Publicado: (2025)

Understanding the Dynamics of Demonstration Conflict in In-Context Learning
por: Jiao, Difan, et al.
Publicado: (2026)

Selective Induction Heads: How Transformers Select Causal Structures In Context
por: D'Angelo, Francesco, et al.
Publicado: (2025)

Does higher interpretability imply better utility? A Pairwise Analysis on Sparse Autoencoders
por: Wang, Xu, et al.
Publicado: (2025)

Multi-Head Attention as a Source of Catastrophic Forgetting in MoE Transformers
por: Chen, Anrui, et al.
Publicado: (2026)

Improving Transformers with Dynamically Composable Multi-Head Attention
por: Xiao, Da, et al.
Publicado: (2024)

Which Attention Heads Matter for In-Context Learning?
por: Yin, Kayo, et al.
Publicado: (2025)