Weak-to-Strong Generalization Even in Random Feature Networks, Provably
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909894533185536 |
|---|---|
| author | Medvedev, Marko Lyu, Kaifeng Yu, Dingli Arora, Sanjeev Li, Zhiyuan Srebro, Nathan |
| author_facet | Medvedev, Marko Lyu, Kaifeng Yu, Dingli Arora, Sanjeev Li, Zhiyuan Srebro, Nathan |
| contents | Weak-to-Strong Generalization (Burns et al., 2024) is the phenomenon whereby a strong student, say GPT-4, learns a task from a weak teacher, say GPT-2, and ends up significantly outperforming the teacher. We show that this phenomenon does not require a strong learner like GPT-4. We consider student and teacher that are random feature models, described by two-layer networks with a random and fixed bottom layer and a trained top layer. A "weak" teacher, with a small number of units (i.e. random features), is trained on the population, and a "strong" student, with a much larger number of units (i.e. random features), is trained only on labels generated by the weak teacher. We demonstrate, prove, and understand how the student can outperform the teacher, even though trained only on data labeled by the teacher. We also explain how such weak-to-strong generalization is enabled by early stopping. Importantly, we also show the quantitative limits of weak-to-strong generalization in this model. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_02877 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Weak-to-Strong Generalization Even in Random Feature Networks, Provably Medvedev, Marko Lyu, Kaifeng Yu, Dingli Arora, Sanjeev Li, Zhiyuan Srebro, Nathan Machine Learning Weak-to-Strong Generalization (Burns et al., 2024) is the phenomenon whereby a strong student, say GPT-4, learns a task from a weak teacher, say GPT-2, and ends up significantly outperforming the teacher. We show that this phenomenon does not require a strong learner like GPT-4. We consider student and teacher that are random feature models, described by two-layer networks with a random and fixed bottom layer and a trained top layer. A "weak" teacher, with a small number of units (i.e. random features), is trained on the population, and a "strong" student, with a much larger number of units (i.e. random features), is trained only on labels generated by the weak teacher. We demonstrate, prove, and understand how the student can outperform the teacher, even though trained only on data labeled by the teacher. We also explain how such weak-to-strong generalization is enabled by early stopping. Importantly, we also show the quantitative limits of weak-to-strong generalization in this model. |
| title | Weak-to-Strong Generalization Even in Random Feature Networks, Provably |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2503.02877 |