Buchumschlag

Spike No More: Stabilizing the Pre-training of Large Language Models

Fuente: arXiv

Gespeichert in:

Bibliographische Detailangaben
Hauptverfasser:	Takase, Sho, Kiyono, Shun, Kobayashi, Sosuke, Suzuki, Jun
Format:	Preprint
Veröffentlicht:	2023
Schlagworte:	Computation and Language Artificial Intelligence
Online-Zugang:	Acceder al recurso
Tags:	Tag hinzufügen Keine Tags, Fügen Sie den ersten Tag hinzu!

Ähnliche Einträge

Pre-training LLM without Learning Rate Decay Enhances Supervised Fine-Tuning
von: Yano, Kazuki, et al.
Veröffentlicht: (2026)

Efficient Construction of Model Family through Progressive Training Using Model Expansion
von: Yano, Kazuki, et al.
Veröffentlicht: (2025)

Self-Translate-Train: Enhancing Cross-Lingual Transfer of Large Language Models via Inherent Capability
von: Ri, Ryokan, et al.
Veröffentlicht: (2024)

Large Vocabulary Size Improves Large Language Models
von: Takase, Sho, et al.
Veröffentlicht: (2024)

Natural Fingerprints of Large Language Models
von: Suzuki, Teppei, et al.
Veröffentlicht: (2025)

Pre-trained Large Language Models for Financial Sentiment Analysis
von: Luo, Wei, et al.
Veröffentlicht: (2024)

Understanding Data Temporality Impact on Large Language Models Pre-training
von: Pilchen, Hippolyte, et al.
Veröffentlicht: (2026)

DataMan: Data Manager for Pre-training Large Language Models
von: Peng, Ru, et al.
Veröffentlicht: (2025)

Revisiting the Capacity Gap in Chain-of-Thought Distillation from a Practical Perspective
von: Kajitsuka, Tokio, et al.
Veröffentlicht: (2026)

Pre-training Distillation for Large Language Models: A Design Space Exploration
von: Peng, Hao, et al.
Veröffentlicht: (2024)

Efficient Language Adaptive Pre-training: Extending State-of-the-Art Large Language Models for Polish
von: Ruciński, Szymon
Veröffentlicht: (2024)

SPAFIT: Stratified Progressive Adaptation Fine-tuning for Pre-trained Large Language Models
von: Arora, Samir, et al.
Veröffentlicht: (2024)

Towards Tracing Trustworthiness Dynamics: Revisiting Pre-training Period of Large Language Models
von: Qian, Chen, et al.
Veröffentlicht: (2024)

Probing Language Models for Pre-training Data Detection
von: Liu, Zhenhua, et al.
Veröffentlicht: (2024)

Synthesize-on-Graph: Knowledgeable Synthetic Data Generation for Continue Pre-training of Large Language Models
von: Ma, Shengjie, et al.
Veröffentlicht: (2025)

MELT: Materials-aware Continued Pre-training for Language Model Adaptation to Materials Science
von: Kim, Junho, et al.
Veröffentlicht: (2024)

Sparse is Enough in Fine-tuning Pre-trained Large Language Models
von: Song, Weixi, et al.
Veröffentlicht: (2023)

Simple and Scalable Strategies to Continually Pre-train Large Language Models
von: Ibrahim, Adam, et al.
Veröffentlicht: (2024)

PreCog: Exploring the Relation between Memorization and Performance in Pre-trained Language Models
von: Ranaldi, Leonardo, et al.
Veröffentlicht: (2023)

Can Pre-trained Language Models Understand Chinese Humor?
von: Chen, Yuyan, et al.
Veröffentlicht: (2024)

How to inject knowledge efficiently? Knowledge Infusion Scaling Law for Pre-training Large Language Models
von: Lv, Kangtao, et al.
Veröffentlicht: (2025)

Machine Unlearning of Pre-trained Large Language Models
von: Yao, Jin, et al.
Veröffentlicht: (2024)

Find Parent then Label Children: A Two-stage Taxonomy Completion Method with Pre-trained Language Model
von: Xia, Fei, et al.
Veröffentlicht: (2024)

Blacks is to Anger as Whites is to Joy? Understanding Latent Affective Bias in Large Pre-trained Neural Language Models
von: Kadan, Anoop, et al.
Veröffentlicht: (2023)

Scaling Smart: Accelerating Large Language Model Pre-training with Small Model Initialization
von: Samragh, Mohammad, et al.
Veröffentlicht: (2024)

From N-grams to Pre-trained Multilingual Models For Language Identification
von: Sindane, Thapelo, et al.
Veröffentlicht: (2024)

Boosting Explainability through Selective Rationalization in Pre-trained Language Models
von: Yuan, Libing, et al.
Veröffentlicht: (2025)

RegMix: Data Mixture as Regression for Language Model Pre-training
von: Liu, Qian, et al.
Veröffentlicht: (2024)

More Women, Same Stereotypes: Unpacking the Gender Bias Paradox in Large Language Models
von: Chen, Evan, et al.
Veröffentlicht: (2025)

DocMamba: Efficient Document Pre-training with State Space Model
von: Hu, Pengfei, et al.
Veröffentlicht: (2024)

Efficient Data Learning for Open Information Extraction with Pre-trained Language Models
von: Fan, Zhiyuan, et al.
Veröffentlicht: (2023)

Adaptive Few-shot Prompting for Machine Translation with Pre-trained Language Models
von: Tang, Lei, et al.
Veröffentlicht: (2025)

Making Metadata More FAIR Using Large Language Models
von: Sundaram, Sowmya S., et al.
Veröffentlicht: (2023)

Sequence-to-Sequence Spanish Pre-trained Language Models
von: Araujo, Vladimir, et al.
Veröffentlicht: (2023)

Investigating Data Contamination for Pre-training Language Models
von: Jiang, Minhao, et al.
Veröffentlicht: (2024)

Aligning Pre-trained Models for Spoken Language Translation
von: Sedláček, Šimon, et al.
Veröffentlicht: (2024)

Topic Over Source: The Key to Effective Data Mixing for Language Models Pre-training
von: Peng, Jiahui, et al.
Veröffentlicht: (2025)

Making Pre-trained Language Models Better Continual Few-Shot Relation Extractors
von: Ma, Shengkun, et al.
Veröffentlicht: (2024)

B-cos LM: Efficiently Transforming Pre-trained Language Models for Improved Explainability
von: Wang, Yifan, et al.
Veröffentlicht: (2025)

A Survey on Post-training of Large Language Models
von: Tie, Guiyao, et al.
Veröffentlicht: (2025)