Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | https://arxiv.org/abs/2505.23337 |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866918226674319360 |
|---|---|
| author | Verma, Chetan Timmaraju, Aditya Srinivas Hsieh, Cho-Jui Damle, Suyash Bui, Ngot Zhang, Yang Chen, Wen Liu, Xin Jain, Prateek Dhillon, Inderjit S |
| author_facet | Verma, Chetan Timmaraju, Aditya Srinivas Hsieh, Cho-Jui Damle, Suyash Bui, Ngot Zhang, Yang Chen, Wen Liu, Xin Jain, Prateek Dhillon, Inderjit S |
| contents | Industry-grade ML models are carefully designed to meet rapidly evolving serving constraints, which requires significant resources for model development. In this paper, we propose MatTA, a framework for training multiple accurate Student models using a novel Teacher-TA-Student recipe. TA models are larger versions of the Student models with higher capacity, and thus allow Student models to better relate to the Teacher model and also bring in more domain-specific expertise. Furthermore, multiple accurate Student models can be extracted from the TA model. Therefore, despite only one training run, our methodology provides multiple servable options to trade off accuracy for lower serving cost. We demonstrate the proposed method, MatTA, on proprietary datasets and models. Its practical efficacy is underscored by live A/B tests within a production ML system, demonstrating 20% improvement on a key metric. We also demonstrate our method on GPT-2 Medium, a public model, and achieve relative improvements of over 24% on SAT Math and over 10% on the LAMBADA benchmark. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_23337 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Matryoshka Model Learning for Improved Elastic Student Models Verma, Chetan Timmaraju, Aditya Srinivas Hsieh, Cho-Jui Damle, Suyash Bui, Ngot Zhang, Yang Chen, Wen Liu, Xin Jain, Prateek Dhillon, Inderjit S Machine Learning Artificial Intelligence Industry-grade ML models are carefully designed to meet rapidly evolving serving constraints, which requires significant resources for model development. In this paper, we propose MatTA, a framework for training multiple accurate Student models using a novel Teacher-TA-Student recipe. TA models are larger versions of the Student models with higher capacity, and thus allow Student models to better relate to the Teacher model and also bring in more domain-specific expertise. Furthermore, multiple accurate Student models can be extracted from the TA model. Therefore, despite only one training run, our methodology provides multiple servable options to trade off accuracy for lower serving cost. We demonstrate the proposed method, MatTA, on proprietary datasets and models. Its practical efficacy is underscored by live A/B tests within a production ML system, demonstrating 20% improvement on a key metric. We also demonstrate our method on GPT-2 Medium, a public model, and achieve relative improvements of over 24% on SAT Math and over 10% on the LAMBADA benchmark. |
| title | Matryoshka Model Learning for Improved Elastic Student Models |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2505.23337 |