Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.07746 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915147534041088 |
|---|---|
| author | Magen, Roey Shang, Shuning Xu, Zhiwei Frei, Spencer Hu, Wei Vardi, Gal |
| author_facet | Magen, Roey Shang, Shuning Xu, Zhiwei Frei, Spencer Hu, Wei Vardi, Gal |
| contents | The phenomenon of benign overfitting, where a trained neural network perfectly fits noisy training data but still achieves near-optimal test performance, has been extensively studied in recent years for linear models and fully-connected/convolutional networks. In this work, we study benign overfitting in a single-head softmax attention model, which is the fundamental building block of Transformers. We prove that under appropriate conditions, the model exhibits benign overfitting in a classification setting already after two steps of gradient descent. Moreover, we show conditions where a minimum-norm/maximum-margin interpolator exhibits benign overfitting. We study how the overfitting behavior depends on the signal-to-noise ratio (SNR) of the data distribution, namely, the ratio between norms of signal and noise tokens, and prove that a sufficiently large SNR is both necessary and sufficient for benign overfitting. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_07746 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Benign Overfitting in Single-Head Attention Magen, Roey Shang, Shuning Xu, Zhiwei Frei, Spencer Hu, Wei Vardi, Gal Machine Learning The phenomenon of benign overfitting, where a trained neural network perfectly fits noisy training data but still achieves near-optimal test performance, has been extensively studied in recent years for linear models and fully-connected/convolutional networks. In this work, we study benign overfitting in a single-head softmax attention model, which is the fundamental building block of Transformers. We prove that under appropriate conditions, the model exhibits benign overfitting in a classification setting already after two steps of gradient descent. Moreover, we show conditions where a minimum-norm/maximum-margin interpolator exhibits benign overfitting. We study how the overfitting behavior depends on the signal-to-noise ratio (SNR) of the data distribution, namely, the ratio between norms of signal and noise tokens, and prove that a sufficiently large SNR is both necessary and sufficient for benign overfitting. |
| title | Benign Overfitting in Single-Head Attention |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2410.07746 |