The Pitfalls of Memorization: When Memorization Hurts Generalization
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913606183944192 |
|---|---|
| author | Bayat, Reza Pezeshki, Mohammad Dohmatob, Elvis Lopez-Paz, David Vincent, Pascal |
| author_facet | Bayat, Reza Pezeshki, Mohammad Dohmatob, Elvis Lopez-Paz, David Vincent, Pascal |
| contents | Neural networks often learn simple explanations that fit the majority of the data while memorizing exceptions that deviate from these explanations.This behavior leads to poor generalization when the learned explanations rely on spurious correlations. In this work, we formalize the interplay between memorization and generalization, showing that spurious correlations would particularly lead to poor generalization when are combined with memorization. Memorization can reduce training loss to zero, leaving no incentive to learn robust, generalizable patterns. To address this, we propose memorization-aware training (MAT), which uses held-out predictions as a signal of memorization to shift a model's logits. MAT encourages learning robust patterns invariant across distributions, improving generalization under distribution shifts. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_07684 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | The Pitfalls of Memorization: When Memorization Hurts Generalization Bayat, Reza Pezeshki, Mohammad Dohmatob, Elvis Lopez-Paz, David Vincent, Pascal Machine Learning Artificial Intelligence Neural networks often learn simple explanations that fit the majority of the data while memorizing exceptions that deviate from these explanations.This behavior leads to poor generalization when the learned explanations rely on spurious correlations. In this work, we formalize the interplay between memorization and generalization, showing that spurious correlations would particularly lead to poor generalization when are combined with memorization. Memorization can reduce training loss to zero, leaving no incentive to learn robust, generalizable patterns. To address this, we propose memorization-aware training (MAT), which uses held-out predictions as a signal of memorization to shift a model's logits. MAT encourages learning robust patterns invariant across distributions, improving generalization under distribution shifts. |
| title | The Pitfalls of Memorization: When Memorization Hurts Generalization |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2412.07684 |