How Well Can Transformers Emulate In-context Newton's Method?

Documenti analoghi

• Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformers
  di: Chen, Siyu, et al.
  Pubblicazione: (2024)
• A Riemannian Optimization Perspective of the Gauss-Newton Method for Feedforward Neural Networks
  di: Cayci, Semih
  Pubblicazione: (2024)
• The Newton-Muon Optimizer
  di: Du, Zhehang, et al.
  Pubblicazione: (2026)
• One-Layer Transformer Provably Learns One-Nearest Neighbor In Context
  di: Li, Zihao, et al.
  Pubblicazione: (2024)
• Implicit Regularization of Gradient Flow on One-Layer Softmax Attention
  di: Sheen, Heejune, et al.
  Pubblicazione: (2024)