In-Context Learning of a Linear Transformer Block: Benefits of the MLP Component and One-Step GD Initialization

Documenti analoghi

• Half the Nonlinearity Is Wasted: Measuring and Reallocating the Transformer's MLP Budget
  di: Balogh, Peter
  Pubblicazione: (2026)
• Interpreting Context Look-ups in Transformers: Investigating Attention-MLP Interactions
  di: Neo, Clement, et al.
  Pubblicazione: (2024)
• Minimax Optimal Convergence of Gradient Descent in Logistic Regression via Large and Adaptive Stepsizes
  di: Zhang, Ruiqi, et al.
  Pubblicazione: (2025)
• How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?
  di: Wu, Jingfeng, et al.
  Pubblicazione: (2023)
• MLP-KAN: Unifying Deep Representation and Function Learning
  di: He, Yunhong, et al.
  Pubblicazione: (2024)