Image de couverture de livre

Scaling Smart: Accelerating Large Language Model Pre-training with Small Model Initialization

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Samragh, Mohammad, Mirzadeh, Iman, Vahid, Keivan Alizadeh, Faghri, Fartash, Cho, Minsik, Nabi, Moin, Naik, Devang, Farajtabar, Mehrdad
Format:	Preprint
Publié:	2024
Sujets:	Computation and Language Artificial Intelligence Machine Learning
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Computational Bottlenecks of Training Small-scale Large Language Models
par: Ashkboos, Saleh, et autres
Publié: (2024)

Barriers for Learning in an Evolving World: Mathematical Understanding of Loss of Plasticity
par: Joudaki, Amir, et autres
Publié: (2025)

Duo-LLM: A Framework for Studying Adaptive Computation in Large Language Models
par: Alizadeh, Keivan, et autres
Publié: (2024)

Recursive Language Models Meet Uncertainty: The Surprising Effectiveness of Self-Reflective Program Search for Long Context
par: Alizadeh, Keivan, et autres
Publié: (2026)

Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potential
par: Samragh, Mohammad, et autres
Publié: (2025)

SALSA: Soup-based Alignment Learning for Stronger Adaptation in RLHF
par: Chegini, Atoosa, et autres
Publié: (2024)

LLM in a flash: Efficient Large Language Model Inference with Limited Memory
par: Alizadeh, Keivan, et autres
Publié: (2023)

From Dense to Dynamic: Token-Difficulty Driven MoEfication of Pre-Trained LLMs
par: Nishu, Kumari, et autres
Publié: (2025)

GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models
par: Mirzadeh, Iman, et autres
Publié: (2024)

The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity
par: Shojaee, Parshin, et autres
Publié: (2025)

Knowledge Transfer from Vision Foundation Models for Efficient Training of Small Task-specific Models
par: Vemulapalli, Raviteja, et autres
Publié: (2023)

KV-Runahead: Scalable Causal LLM Inference by Parallel Key-Value Cache Generation
par: Cho, Minsik, et autres
Publié: (2024)

CatLIP: CLIP-level Visual Recognition Accuracy with 2.7x Faster Pre-training on Web-scale Image-Text Data
par: Mehta, Sachin, et autres
Publié: (2024)

TiC-LM: A Web-Scale Benchmark for Time-Continual LLM Pretraining
par: Li, Jeffrey, et autres
Publié: (2025)

SLiCK: Exploiting Subsequences for Length-Constrained Keyword Spotting
par: Nishu, Kumari, et autres
Publié: (2024)

MoE-PHDS: One MoE checkpoint for flexible runtime sparsity
par: Hannah, Lauren. A, et autres
Publié: (2025)

Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Why
par: Armandpour, Mohammadreza, et autres
Publié: (2026)

TiC-CLIP: Continual Training of CLIP Models
par: Garg, Saurabh, et autres
Publié: (2023)

SpecMD: A Comprehensive Study On Speculative Expert Prefetching
par: Hoang, Duc, et autres
Publié: (2026)

Improving vision-inspired keyword spotting using dynamic module skipping in streaming conformer encoder
par: Bittar, Alexandre, et autres
Publié: (2023)

SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understanding
par: Wang, Haoxiang, et autres
Publié: (2023)

SPD: Sync-Point Drop for Efficient Tensor Parallelism of Large Language Models
par: Kim, Han-Byul, et autres
Publié: (2025)

Proxy-FDA: Proxy-based Feature Distribution Alignment for Fine-tuning Vision Foundation Models without Forgetting
par: Huang, Chen, et autres
Publié: (2025)

MoEs Are Stronger than You Think: Hyper-Parallel Inference Scaling with RoE
par: Zibakhsh, Soheil, et autres
Publié: (2025)

TIDE: Every Layer Knows the Token Beneath the Context
par: Jaiswal, Ajay, et autres
Publié: (2026)

Contrastive Perplexity for Controlled Generation: An Application in Detoxifying Large Language Models
par: Klein, Tassilo, et autres
Publié: (2024)

MemoryLLM: Plug-n-Play Interpretable Feed-Forward Memory for Transformers
par: Jaiswal, Ajay, et autres
Publié: (2026)

MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training
par: Vasu, Pavan Kumar Anasosalu, et autres
Publié: (2023)

Learning Private Representations through Entropy-based Adversarial Training
par: Klein, Tassilo, et autres
Publié: (2025)

CLIP with Quality Captions: A Strong Pretraining for Vision Tasks
par: Vasu, Pavan Kumar Anasosalu, et autres
Publié: (2024)

SpecHop: Continuous Speculation for Accelerating Multi-Hop Retrieval Agents
par: Saberi, Mehrdad, et autres
Publié: (2026)

RepCNN: Micro-sized, Mighty Models for Wakeword Detection
par: Kundu, Arnav, et autres
Publié: (2024)

MUSCLE: A Model Update Strategy for Compatible LLM Evolution
par: Echterhoff, Jessica, et autres
Publié: (2024)

The path towards contact-based physical human-robot interaction
par: Farajtabar, Mohammad, et autres
Publié: (2024)

Utilization of Pre-trained Language Model for Adapter-based Knowledge Transfer in Software Engineering
par: Saberi, Iman, et autres
Publié: (2023)

Revisiting the Past: Data Unlearning with Model State History
par: Rezaei, Keivan, et autres
Publié: (2025)

Metadata Conditioning Accelerates Language Model Pre-training
par: Gao, Tianyu, et autres
Publié: (2025)

On Initializing Transformers with Pre-trained Embeddings
par: Kim, Ha Young, et autres
Publié: (2024)

Don't Pay Attention
par: Hammoud, Mohammad, et autres
Publié: (2025)

Avey-B
par: Acharya, Devang, et autres
Publié: (2026)