Cover Image

On the Optimization and Generalization of Multi-head Attention

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Deora, Puneesh, Ghaderi, Rouzbeh, Taheri, Hossein, Thrampoulidis, Christos
Format:	Preprint
Published:	2023
Subjects:	Machine Learning Optimization and Control
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Implicit Bias and Fast Convergence Rates for Self-attention
by: Vasudeva, Bhavya, et al.
Published: (2024)

Facts in Stats: Impacts of Pretraining Diversity on Language Model Generalization
by: Behnia, Tina, et al.
Published: (2025)

Implicit Bias of Spectral Descent and Muon on Multiclass Separable Data
by: Fan, Chen, et al.
Published: (2025)

Diagonalizing the Softmax: Hadamard Initialization for Tractable Cross-Entropy Dynamics
by: Garrod, Connall, et al.
Published: (2025)

In-Context Occam's Razor: How Transformers Prefer Simpler Hypotheses on the Fly
by: Deora, Puneesh, et al.
Published: (2025)

How Muon's Spectral Design Benefits Generalization: A Study on Imbalanced Data
by: Vasudeva, Bhavya, et al.
Published: (2025)

Memory capacity of two layer neural networks with smooth activations
by: Madden, Liam, et al.
Published: (2023)

Understanding Contextual Recall in Transformers: How Finetuning Enables In-Context Reasoning over Pretraining Knowledge
by: Vasudeva, Bhavya, et al.
Published: (2026)

Next-token prediction capacity: general upper bounds and a lower bound for transformers
by: Madden, Liam, et al.
Published: (2024)

Transformers as Support Vector Machines
by: Tarzanagh, Davoud Ataee, et al.
Published: (2023)

On Adaptivity in Zeroth-Order Optimization
by: Dbouk, Hassan, et al.
Published: (2026)

Attention is All You Need to Optimize Wind Farm Operations and Maintenance
by: Kazemian, Iman, et al.
Published: (2024)

Intersection of Reinforcement Learning and Bayesian Optimization for Intelligent Control of Industrial Processes: A Safe MPC-based DPG using Multi-Objective BO
by: Esfahani, Hossein Nejatbakhsh, et al.
Published: (2025)

Multi-Task Learning Based on Support Vector Machines and Twin Support Vector Machines: A Comprehensive Survey
by: Bazikar, Fatemeh, et al.
Published: (2025)

Aligned Multi Objective Optimization
by: Efroni, Yonathan, et al.
Published: (2025)

Accelerating Multi-Block Constrained Optimization Through Learning to Optimize
by: Liang, Ling, et al.
Published: (2024)

Adaptive Smooth Tchebycheff Attention for Multi-Objective Policy Optimization
by: Murillo-Gonzalez, Alejandro, et al.
Published: (2026)

Sharper Guarantees for Learning Neural Network Classifiers with Gradient Methods
by: Taheri, Hossein, et al.
Published: (2024)

Distributionally Robust Multi-Objective Optimization
by: Yang, Yufeng, et al.
Published: (2026)

Preferential Multi-Objective Bayesian Optimization
by: Astudillo, Raul, et al.
Published: (2024)

Safe Online Convex Optimization with Multi-Point Feedback
by: Hutchinson, Spencer, et al.
Published: (2024)

MOSS: Multi-Objective Optimization for Stable Rule Sets
by: Liu, Brian, et al.
Published: (2025)

Multi-fidelity Machine Learning for Uncertainty Quantification and Optimization
by: Zhang, Ruda, et al.
Published: (2024)

A General and Streamlined Differentiable Optimization Framework
by: Rosemberg, Andrew W., et al.
Published: (2025)

Tight Generalization Bounds for Noiseless Inverse Optimization
by: Fatemi, Pouria, et al.
Published: (2026)

Knowledge Gradient for Multi-Objective Bayesian Optimization with Decoupled Evaluations
by: Buckingham, Jack M., et al.
Published: (2023)

Non-linear Multi-objective Optimization with Probabilistic Branch and Bound
by: Huang, Hao, et al.
Published: (2025)

Attention-based PCA
by: Maulen-Soto, Rodrigo, et al.
Published: (2026)

CLion: Efficient Cautious Lion Optimizer with Enhanced Generalization
by: Huang, Feihu, et al.
Published: (2026)

On Generalization and Regularization via Wasserstein Distributionally Robust Optimization
by: Wu, Qinyu, et al.
Published: (2022)

Controllable Expensive Multi-objective Learning with Warm-starting Bayesian Optimization
by: Nguyen, Quang-Huy, et al.
Published: (2023)

Multi-Objective Optimization via Wasserstein-Fisher-Rao Gradient Flow
by: Ren, Yinuo, et al.
Published: (2023)

Multi-Timescale Primal Dual Hybrid Gradient with Application to Distributed Optimization
by: Zhang, Junhui, et al.
Published: (2025)

A Multi-objective Newton Optimization Algorithm for Hyper-Parameter Search
by: Xu, Qinwu
Published: (2024)

Collaborative Pareto Set Learning in Multiple Multi-Objective Optimization Problems
by: Shang, Chikai, et al.
Published: (2024)

Physics-Aware Multifidelity Bayesian Optimization: a Generalized Formulation
by: Di Fiore, Francesco, et al.
Published: (2023)

Enhancing Multi-Objective Optimization through Machine Learning-Supported Multiphysics Simulation
by: Botache, Diego, et al.
Published: (2023)

Data-driven Mixed Integer Optimization through Probabilistic Multi-variable Branching
by: Chen, Yanguang, et al.
Published: (2023)

Multi-Objective Optimization-Based Anonymization of Structured Data for Machine Learning Application
by: Wei, Yusi, et al.
Published: (2025)

Revisiting Multi-Agent Asynchronous Online Optimization with Delays: the Strongly Convex Case
by: Bao, Lingchan, et al.
Published: (2025)