Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2507.07814 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916837256593408 |
|---|---|
| author | Yudin, Nikolay Gaponov, Alexander Kudriashov, Sergei Rakhuba, Maxim |
| author_facet | Yudin, Nikolay Gaponov, Alexander Kudriashov, Sergei Rakhuba, Maxim |
| contents | We present a novel local Lipschitz bound for self-attention blocks of transformers. This bound is based on a refined closed-form expression for the spectral norm of the softmax function. The resulting bound is not only more accurate than in the prior art, but also unveils the dependence of the Lipschitz constant on attention score maps. Based on the new findings, we suggest an explanation of the way distributions inside the attention map affect the robustness from the Lipschitz constant perspective. We also introduce a new lightweight regularization term called JaSMin (Jacobian Softmax norm Minimization), which boosts the transformer's robustness and decreases local Lipschitz constants of the whole network. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_07814 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Pay Attention to Attention Distribution: A New Local Lipschitz Bound for Transformers Yudin, Nikolay Gaponov, Alexander Kudriashov, Sergei Rakhuba, Maxim Machine Learning Numerical Analysis 15A42, 15A60, 68T07 We present a novel local Lipschitz bound for self-attention blocks of transformers. This bound is based on a refined closed-form expression for the spectral norm of the softmax function. The resulting bound is not only more accurate than in the prior art, but also unveils the dependence of the Lipschitz constant on attention score maps. Based on the new findings, we suggest an explanation of the way distributions inside the attention map affect the robustness from the Lipschitz constant perspective. We also introduce a new lightweight regularization term called JaSMin (Jacobian Softmax norm Minimization), which boosts the transformer's robustness and decreases local Lipschitz constants of the whole network. |
| title | Pay Attention to Attention Distribution: A New Local Lipschitz Bound for Transformers |
| topic | Machine Learning Numerical Analysis 15A42, 15A60, 68T07 |
| url | https://arxiv.org/abs/2507.07814 |