LLM Pruning and Distillation in Practice: The Minitron Approach
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913601813479424 |
|---|---|
| author | Sreenivas, Sharath Turuvekere Muralidharan, Saurav Joshi, Raviraj Chochowski, Marcin Mahabaleshwarkar, Ameya Sunil Shen, Gerald Zeng, Jiaqi Chen, Zijia Suhara, Yoshi Diao, Shizhe Yu, Chenhan Chen, Wei-Chun Ross, Hayley Olabiyi, Oluwatobi Aithal, Ashwath Kuchaiev, Oleksii Korzekwa, Daniel Molchanov, Pavlo Patwary, Mostofa Shoeybi, Mohammad Kautz, Jan Catanzaro, Bryan |
| author_facet | Sreenivas, Sharath Turuvekere Muralidharan, Saurav Joshi, Raviraj Chochowski, Marcin Mahabaleshwarkar, Ameya Sunil Shen, Gerald Zeng, Jiaqi Chen, Zijia Suhara, Yoshi Diao, Shizhe Yu, Chenhan Chen, Wei-Chun Ross, Hayley Olabiyi, Oluwatobi Aithal, Ashwath Kuchaiev, Oleksii Korzekwa, Daniel Molchanov, Pavlo Patwary, Mostofa Shoeybi, Mohammad Kautz, Jan Catanzaro, Bryan |
| contents | We present a comprehensive report on compressing the Llama 3.1 8B and Mistral NeMo 12B models to 4B and 8B parameters, respectively, using pruning and distillation. We explore two distinct pruning strategies: (1) depth pruning and (2) joint hidden/attention/MLP (width) pruning, and evaluate the results on common benchmarks from the LM Evaluation Harness. The models are then aligned with NeMo Aligner and tested in instruct-tuned versions. This approach produces a compelling 4B model from Llama 3.1 8B and a state-of-the-art Mistral-NeMo-Minitron-8B (MN-Minitron-8B for brevity) model from Mistral NeMo 12B. We found that with no access to the original data, it is beneficial to slightly fine-tune teacher models on the distillation dataset. We open-source our base model weights on Hugging Face with a permissive license. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_11796 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LLM Pruning and Distillation in Practice: The Minitron Approach Sreenivas, Sharath Turuvekere Muralidharan, Saurav Joshi, Raviraj Chochowski, Marcin Mahabaleshwarkar, Ameya Sunil Shen, Gerald Zeng, Jiaqi Chen, Zijia Suhara, Yoshi Diao, Shizhe Yu, Chenhan Chen, Wei-Chun Ross, Hayley Olabiyi, Oluwatobi Aithal, Ashwath Kuchaiev, Oleksii Korzekwa, Daniel Molchanov, Pavlo Patwary, Mostofa Shoeybi, Mohammad Kautz, Jan Catanzaro, Bryan Computation and Language Artificial Intelligence Machine Learning We present a comprehensive report on compressing the Llama 3.1 8B and Mistral NeMo 12B models to 4B and 8B parameters, respectively, using pruning and distillation. We explore two distinct pruning strategies: (1) depth pruning and (2) joint hidden/attention/MLP (width) pruning, and evaluate the results on common benchmarks from the LM Evaluation Harness. The models are then aligned with NeMo Aligner and tested in instruct-tuned versions. This approach produces a compelling 4B model from Llama 3.1 8B and a state-of-the-art Mistral-NeMo-Minitron-8B (MN-Minitron-8B for brevity) model from Mistral NeMo 12B. We found that with no access to the original data, it is beneficial to slightly fine-tune teacher models on the distillation dataset. We open-source our base model weights on Hugging Face with a permissive license. |
| title | LLM Pruning and Distillation in Practice: The Minitron Approach |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2408.11796 |