Image de couverture de livre

LLM Pruning and Distillation in Practice: The Minitron Approach

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Sreenivas, Sharath Turuvekere, Muralidharan, Saurav, Joshi, Raviraj, Chochowski, Marcin, Mahabaleshwarkar, Ameya Sunil, Shen, Gerald, Zeng, Jiaqi, Chen, Zijia, Suhara, Yoshi, Diao, Shizhe, Yu, Chenhan, Chen, Wei-Chun, Ross, Hayley, Olabiyi, Oluwatobi, Aithal, Ashwath, Kuchaiev, Oleksii, Korzekwa, Daniel, Molchanov, Pavlo, Patwary, Mostofa, Shoeybi, Mohammad, Kautz, Jan, Catanzaro, Bryan
Format:	Preprint
Publié:	2024
Sujets:	Computation and Language Artificial Intelligence Machine Learning
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Minitron-SSM: Efficient Hybrid Language Model Compression through Group-Aware SSM Pruning
par: Taghibakhshi, Ali, et autres
Publié: (2025)

Nemotron Elastic: Towards Efficient Many-in-One Reasoning LLMs
par: Taghibakhshi, Ali, et autres
Publié: (2025)

Compact Language Models via Pruning and Knowledge Distillation
par: Muralidharan, Saurav, et autres
Publié: (2024)

Star Elastic: Many-in-One Reasoning LLMs with Efficient Budget Control
par: Taghibakhshi, Ali, et autres
Publié: (2026)

X-Token: Projection-Guided Cross-Tokenizer Knowledge Distillation
par: Sreenivas, Sharath Turuvekere, et autres
Publié: (2026)

When2Call: When (not) to Call Tools
par: Ross, Hayley, et autres
Publié: (2025)

Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Models
par: Parmar, Jupinder, et autres
Publié: (2024)

Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training
par: Diao, Shizhe, et autres
Publié: (2025)

Hymba: A Hybrid-head Architecture for Small Language Models
par: Dong, Xin, et autres
Publié: (2024)

Nemotron-CC-Math: A 133 Billion-Token-Scale High Quality Math Pretraining Dataset
par: Mahabadi, Rabeeh Karimi, et autres
Publié: (2025)

Maximize Your Data's Potential: Enhancing LLM Accuracy with Two-Phase Pretraining
par: Feng, Steven, et autres
Publié: (2024)

Quantization-Aware Distillation for NVFP4 Inference Accuracy Recovery
par: Xin, Meng, et autres
Publié: (2026)

Front-Loading Reasoning: The Synergy between Pretraining and Post-Training Data
par: Akter, Syeda Nahida, et autres
Publié: (2025)

Scaling Fine-Grained MoE Beyond 50B Parameters: Empirical Evaluation and Practical Insights
par: Krajewski, Jakub, et autres
Publié: (2025)

FusionFactory: Fusing LLM Capabilities with Multi-LLM Log Data
par: Feng, Tao, et autres
Publié: (2025)

Source Identification in Abstractive Summarization
par: Suhara, Yoshi, et autres
Publié: (2024)

Nemotron-4 15B Technical Report
par: Parmar, Jupinder, et autres
Publié: (2024)

MIND: Math Informed syNthetic Dialogues for Pretraining LLMs
par: Akter, Syeda Nahida, et autres
Publié: (2024)

RLP: Reinforcement as a Pretraining Objective
par: Hatamizadeh, Ali, et autres
Publié: (2025)

Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset
par: Su, Dan, et autres
Publié: (2024)

Data, Data Everywhere: A Guide for Pretraining Dataset Construction
par: Parmar, Jupinder, et autres
Publié: (2024)

Upcycling Large Language Models into Mixture of Experts
par: He, Ethan, et autres
Publié: (2024)

Noisy Pairing and Partial Supervision for Stylized Opinion Summarization
par: Iso, Hayate, et autres
Publié: (2022)

Large Language Models are Inconsistent and Biased Evaluators
par: Stureborg, Rickard, et autres
Publié: (2024)

Retro-Search: Exploring Untaken Paths for Deeper and Efficient Reasoning
par: Lu, Ximing, et autres
Publié: (2025)

Nemotron-CrossThink: Scaling Self-Learning beyond Math Reasoning
par: Akter, Syeda Nahida, et autres
Publié: (2025)

Your Large Language Models Are Leaving Fingerprints
par: McGovern, Hope, et autres
Publié: (2024)

Small Language Models are the Future of Agentic AI
par: Belcak, Peter, et autres
Publié: (2025)

Prismatic Synthesis: Gradient-based Data Diversification Boosts Generalization in LLM Reasoning
par: Jung, Jaehun, et autres
Publié: (2025)

Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on Diverse Video Games
par: Park, Dongmin, et autres
Publié: (2025)

Optimization of Wind Turbine Placement for Maximum Energy Output in Bangladesh
par: Winner, Olabiyi
Publié: (2026)

Policy and Investment Strategies for Expanding Wind Energy Projects in Bangladesh
par: Winner, Olabiyi
Publié: (2026)

Potential Blind Spots and Hot Spots for Nutrient Loss
par: Kaine Korzekwa
Publié: (2024)

Evaluating Overhead Irrigation Sprinkler Packages in Utah
par: Kaine Korzekwa
Publié: (2024)

AceMath: Advancing Frontier Math Reasoning with Post-Training and Reward Modeling
par: Liu, Zihan, et autres
Publié: (2024)

Multiplication Operator Semigroups on Banach lattice valued continuous function spaces
par: Olabiyi, Tobi David
Publié: (2025)

NeMo-Aligner: Scalable Toolkit for Efficient Model Alignment
par: Shen, Gerald, et autres
Publié: (2024)

On Data Engineering for Scaling LLM Terminal Capabilities
par: Pi, Renjie, et autres
Publié: (2026)

Anonymous Survey on On-demand e-exams WS 2024-25
par: Patwary, Mozaher
Publié: (2026)

Llama 3 Meets MoE: Efficient Upcycling
par: Vavre, Aditya, et autres
Publié: (2024)