Image de couverture de livre

Domain-Adaptive Continued Pre-Training of Small Language Models

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteur principal:	Faroz, Salman
Format:	Preprint
Publié:	2025
Sujets:	Computation and Language Machine Learning
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

DisEmbed: Transforming Disease Understanding through Embeddings
par: Faroz, Salman
Publié: (2024)

Mortgage Language Model: Domain-Adaptive Pretraining with Residual Instruction, Alignment Tuning, and Task-Specific Routing
par: Jain, Manish, et autres
Publié: (2025)

DACP: Domain-Adaptive Continual Pre-Training of Large Language Models for Phone Conversation Summarization
par: Fu, Xue-Yong, et autres
Publié: (2025)

Sensitivity Analysis On Loss Landscape
par: Faroz, Salman
Publié: (2024)

Domain-Adaptive Small Language Models for Structured Tax Code Prediction
par: Nath, Souvik, et autres
Publié: (2025)

Learning Dynamics in Continual Pre-Training for Large Language Models
par: Wang, Xingjin, et autres
Publié: (2025)

On Domain-Adaptive Post-Training for Multimodal Large Language Models
par: Cheng, Daixuan, et autres
Publié: (2024)

Revisiting Replay and Gradient Alignment for Continual Pre-Training of Large Language Models
par: Abbes, Istabrak, et autres
Publié: (2025)

Adapt-Pruner: Adaptive Structural Pruning for Efficient Small Language Model Training
par: Pan, Rui, et autres
Publié: (2025)

ZClip: Adaptive Spike Mitigation for LLM Pre-Training
par: Kumar, Abhay, et autres
Publié: (2025)

Development of Pre-Trained Transformer-based Models for the Nepali Language
par: Thapa, Prajwal, et autres
Publié: (2024)

Hephaestus: Improving Fundamental Agent Capabilities of Large Language Models through Continual Pre-Training
par: Zhuang, Yuchen, et autres
Publié: (2025)

Learning Beyond the Surface: How Far Can Continual Pre-Training with LoRA Enhance LLMs' Domain-Specific Insight Learning?
par: Pezeshkpour, Pouya, et autres
Publié: (2025)

Domain Adaptation of Llama3-70B-Instruct through Continual Pre-Training and Model Merging: A Comprehensive Evaluation
par: Siriwardhana, Shamane, et autres
Publié: (2024)

Safe Reinforcement Learning with Free-form Natural Language Constraints and Pre-Trained Language Models
par: Lou, Xingzhou, et autres
Publié: (2024)

Domain-Adaptive Pre-Training for Arabic Aspect-Based Sentiment Analysis: A Comparative Study of Domain Adaptation and Fine-Tuning Strategies
par: Alyami, Salha, et autres
Publié: (2025)

MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
par: Hu, Shengding, et autres
Publié: (2024)

PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs
par: van der Wal, Oskar, et autres
Publié: (2025)

Empirical Analysis of Efficient Fine-Tuning Methods for Large Pre-Trained Language Models
par: Doering, Nigel, et autres
Publié: (2024)

Efficient Continual Pre-training of LLMs for Low-resource Languages
par: Nag, Arijit, et autres
Publié: (2024)

Domain-Adapted Small Language Models for Reliable Clinical Triage
par: Aljohani, Manar, et autres
Publié: (2026)

Min-K%++: Improved Baseline for Detecting Pre-Training Data from Large Language Models
par: Zhang, Jingyang, et autres
Publié: (2024)

CMR Scaling Law: Predicting Critical Mixture Ratios for Continual Pre-training of Language Models
par: Gu, Jiawei, et autres
Publié: (2024)

Pre-Trained Policy Discriminators are General Reward Models
par: Dou, Shihan, et autres
Publié: (2025)

APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inference
par: Zhao, Bowen, et autres
Publié: (2024)

AGGC: Adaptive Group Gradient Clipping for Stabilizing Large Language Model Training
par: Li, Zhiyuan, et autres
Publié: (2026)

Scaling Smart: Accelerating Large Language Model Pre-training with Small Model Initialization
par: Samragh, Mohammad, et autres
Publié: (2024)

ForeCite: Adapting Pre-Trained Language Models to Predict Future Citation Rates of Academic Papers
par: Hull, Gavin, et autres
Publié: (2025)

Comparative Study of Pre-Trained BERT and Large Language Models for Code-Mixed Named Entity Recognition
par: Shirke, Mayur, et autres
Publié: (2025)

Chinese MentalBERT: Domain-Adaptive Pre-training on Social Media for Chinese Mental Health Text Analysis
par: Zhai, Wei, et autres
Publié: (2024)

Simple and Scalable Strategies to Continually Pre-train Large Language Models
par: Ibrahim, Adam, et autres
Publié: (2024)

CAST: Continuous and Differentiable Semi-Structured Sparsity-Aware Training for Large Language Models
par: Huang, Weiyu, et autres
Publié: (2025)

Robust Pre-Training of Medical Vision-and-Language Models with Domain-Invariant Multi-Modal Masked Reconstruction
par: Filvantorkaman, Melika, et autres
Publié: (2026)

Self-Training for Sample-Efficient Active Learning for Text Classification with Pre-Trained Language Models
par: Schröder, Christopher, et autres
Publié: (2024)

When Does Metadata Conditioning (NOT) Work for Language Model Pre-Training? A Study with Context-Free Grammars
par: Higuchi, Rei, et autres
Publié: (2025)

Data Mixing Agent: Learning to Re-weight Domains for Continual Pre-training
par: Yang, Kailai, et autres
Publié: (2025)

A Post-Training Enhanced Optimization Approach for Small Language Models
par: Zhai, Keke
Publié: (2024)

Train Small, Infer Large: Memory-Efficient LoRA Training for Large Language Models
par: Zhang, Jun, et autres
Publié: (2025)

Model Merging in Pre-training of Large Language Models
par: Li, Yunshui, et autres
Publié: (2025)

Optimizing Pre-Training Data Mixtures with Mixtures of Data Expert Models
par: Belenki, Lior, et autres
Publié: (2025)