Why Fine-grained Labels in Pretraining Benefit Generalization?
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909423045181440 |
|---|---|
| author | Hong, Guan Zhe Cui, Yin Fuxman, Ariel Chan, Stanley Luo, Enming |
| author_facet | Hong, Guan Zhe Cui, Yin Fuxman, Ariel Chan, Stanley Luo, Enming |
| contents | Recent studies show that pretraining a deep neural network with fine-grained labeled data, followed by fine-tuning on coarse-labeled data for downstream tasks, often yields better generalization than pretraining with coarse-labeled data. While there is ample empirical evidence supporting this, the theoretical justification remains an open problem. This paper addresses this gap by introducing a "hierarchical multi-view" structure to confine the input data distribution. Under this framework, we prove that: 1) coarse-grained pretraining only allows a neural network to learn the common features well, while 2) fine-grained pretraining helps the network learn the rare features in addition to the common ones, leading to improved accuracy on hard downstream test samples. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_23129 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Why Fine-grained Labels in Pretraining Benefit Generalization? Hong, Guan Zhe Cui, Yin Fuxman, Ariel Chan, Stanley Luo, Enming Machine Learning Computer Vision and Pattern Recognition Recent studies show that pretraining a deep neural network with fine-grained labeled data, followed by fine-tuning on coarse-labeled data for downstream tasks, often yields better generalization than pretraining with coarse-labeled data. While there is ample empirical evidence supporting this, the theoretical justification remains an open problem. This paper addresses this gap by introducing a "hierarchical multi-view" structure to confine the input data distribution. Under this framework, we prove that: 1) coarse-grained pretraining only allows a neural network to learn the common features well, while 2) fine-grained pretraining helps the network learn the rare features in addition to the common ones, leading to improved accuracy on hard downstream test samples. |
| title | Why Fine-grained Labels in Pretraining Benefit Generalization? |
| topic | Machine Learning Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2410.23129 |