Attention Sinks Are Provably Necessary in Softmax Transformers: Evidence from Trigger-Conditional Tasks

Documenti analoghi

• A Mechanistic Account of Attention Sinks in GPT-2: One Circuit, Broader Implications for Mitigation
  di: Ran-Milo, Yuval, et al.
  Pubblicazione: (2026)
• Outcome-Based RL Provably Leads Transformers to Reason, but Only With the Right Data
  di: Ran-Milo, Yuval, et al.
  Pubblicazione: (2026)
• Softpick: No Attention Sink, No Massive Activations with Rectified Softmax
  di: Zuhri, Zayd M. K., et al.
  Pubblicazione: (2025)
• Minimalist Softmax Attention Provably Learns Constrained Boolean Functions
  di: Hu, Jerry Yao-Chieh, et al.
  Pubblicazione: (2025)
• Provable Benefits of Complex Parameterizations for Structured State Space Models
  di: Ran-Milo, Yuval, et al.
  Pubblicazione: (2024)