Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2402.02769 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913568187744256 |
|---|---|
| author | Jin, Can Che, Tong Peng, Hongwu Li, Yiyuan Metaxas, Dimitris N. Pavone, Marco |
| author_facet | Jin, Can Che, Tong Peng, Hongwu Li, Yiyuan Metaxas, Dimitris N. Pavone, Marco |
| contents | Generalization remains a central challenge in machine learning. In this work, we propose Learning from Teaching (LoT), a novel regularization technique for deep neural networks to enhance generalization. Inspired by the human ability to capture concise and abstract patterns, we hypothesize that generalizable correlations are expected to be easier to imitate. LoT operationalizes this concept to improve the generalization of the main model with auxiliary student learners. The student learners are trained by the main model and, in turn, provide feedback to help the main model capture more generalizable and imitable correlations. Our experimental results across several domains, including Computer Vision, Natural Language Processing, and methodologies like Reinforcement Learning, demonstrate that the introduction of LoT brings significant benefits compared to training models on the original dataset. The results suggest the effectiveness and efficiency of LoT in identifying generalizable information at the right scales while discarding spurious data correlations, thus making LoT a valuable addition to current machine learning. Code is available at https://github.com/jincan333/LoT. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_02769 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Learning from Teaching Regularization: Generalizable Correlations Should be Easy to Imitate Jin, Can Che, Tong Peng, Hongwu Li, Yiyuan Metaxas, Dimitris N. Pavone, Marco Machine Learning Artificial Intelligence Generalization remains a central challenge in machine learning. In this work, we propose Learning from Teaching (LoT), a novel regularization technique for deep neural networks to enhance generalization. Inspired by the human ability to capture concise and abstract patterns, we hypothesize that generalizable correlations are expected to be easier to imitate. LoT operationalizes this concept to improve the generalization of the main model with auxiliary student learners. The student learners are trained by the main model and, in turn, provide feedback to help the main model capture more generalizable and imitable correlations. Our experimental results across several domains, including Computer Vision, Natural Language Processing, and methodologies like Reinforcement Learning, demonstrate that the introduction of LoT brings significant benefits compared to training models on the original dataset. The results suggest the effectiveness and efficiency of LoT in identifying generalizable information at the right scales while discarding spurious data correlations, thus making LoT a valuable addition to current machine learning. Code is available at https://github.com/jincan333/LoT. |
| title | Learning from Teaching Regularization: Generalizable Correlations Should be Easy to Imitate |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2402.02769 |