Cover Image

Drop Dropout on Single-Epoch Language Model Pretraining

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Liu, Houjun, Bauer, John, Manning, Christopher D.
Format:	Preprint
Published:	2025
Subjects:	Computation and Language Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Thoughtbubbles: an Unsupervised Method for Parallel Thinking in Latent Space
by: Liu, Houjun, et al.
Published: (2025)

Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddings
by: Gelberg, Yoav, et al.
Published: (2025)

Layer-wise Regularized Dropout for Neural Language Models
by: Ni, Shiwen, et al.
Published: (2024)

Rethinking the Role of Text Complexity in Language Model Pretraining
by: Velasco, Dan John, et al.
Published: (2025)

MrT5: Dynamic Token Merging for Efficient Byte-level Language Models
by: Kallini, Julie, et al.
Published: (2024)

Assessing The Potential Of Mid-Sized Language Models For Clinical QA
by: Bolton, Elliot, et al.
Published: (2024)

Multilingual Pretraining for Pixel Language Models
by: Kesen, Ilker, et al.
Published: (2025)

ReFT: Representation Finetuning for Language Models
by: Wu, Zhengxuan, et al.
Published: (2024)

Direct Preference Optimization: Your Language Model is Secretly a Reward Model
by: Rafailov, Rafael, et al.
Published: (2023)

Polysemantic Dropout: Conformal OOD Detection for Specialized LLMs
by: Gupta, Ayush, et al.
Published: (2025)

Exploring Polyglot Harmony: On Multilingual Data Allocation for Large Language Models Pretraining
by: Guo, Ping, et al.
Published: (2025)

On Linear Representations and Pretraining Data Frequency in Language Models
by: Merullo, Jack, et al.
Published: (2025)

Gender Encoding Patterns in Pretrained Language Model Representations
by: Zakizadeh, Mahdi, et al.
Published: (2025)

Deciphering the Impact of Pretraining Data on Large Language Models through Machine Unlearning
by: Zhao, Yang, et al.
Published: (2024)

Hallucination Detox: Sensitivity Dropout (SenD) for Large Language Model Training
by: Mohammadzadeh, Shahrad, et al.
Published: (2024)

PonderLM: Pretraining Language Models to Ponder in Continuous Space
by: Zeng, Boyi, et al.
Published: (2025)

Code Pretraining Improves Entity Tracking Abilities of Language Models
by: Kim, Najoung, et al.
Published: (2024)

Emergent Abilities of Large Language Models under Continued Pretraining for Language Adaptation
by: Elhady, Ahmed, et al.
Published: (2025)

Chinese Tiny LLM: Pretraining a Chinese-Centric Large Language Model
by: Du, Xinrun, et al.
Published: (2024)

Enhancing Coreference Resolution with Pretrained Language Models: Bridging the Gap Between Syntax and Semantics
by: Liu, Xingzu, et al.
Published: (2025)

CodePMP: Scalable Preference Model Pretraining for Large Language Model Reasoning
by: Yu, Huimu, et al.
Published: (2024)

A Causal Language Modeling Detour Improves Encoder Continued Pretraining
by: Touchent, Rian, et al.
Published: (2026)

Exploring the Benefits of Domain-Pretraining of Generative Large Language Models for Chemistry
by: Acharya, Anurag, et al.
Published: (2024)

Generative Pretrained Structured Transformers: Unsupervised Syntactic Language Models at Scale
by: Hu, Xiang, et al.
Published: (2024)

Vocabulary Dropout for Curriculum Diversity in LLM Co-Evolution
by: Dineen, Jacob, et al.
Published: (2026)

Label Drop for Multi-Aspect Relation Modeling in Universal Information Extraction
by: Yang, Lu, et al.
Published: (2025)

CCI4.0: A Bilingual Pretraining Dataset for Enhancing Reasoning in Large Language Models
by: Liu, Guang, et al.
Published: (2025)

Memory Decoder: A Pretrained, Plug-and-Play Memory for Large Language Models
by: Cao, Jiaqi, et al.
Published: (2025)

Beyond Random Sampling: Efficient Language Model Pretraining via Curriculum Learning
by: Zhang, Yang, et al.
Published: (2025)

Systematic Analysis for Pretrained Language Model Priming for Parameter-Efficient Fine-tuning
by: Huang, Shih-Cheng, et al.
Published: (2022)

Foundation CAN LM: A Pretrained Language Model For Automotive CAN Data
by: Esashi, Akiharu, et al.
Published: (2026)

AraPoemBERT: A Pretrained Language Model for Arabic Poetry Analysis
by: Qarah, Faisal
Published: (2024)

SaudiBERT: A Large Language Model Pretrained on Saudi Dialect Corpora
by: Qarah, Faisal
Published: (2024)

Harnessing the Intrinsic Knowledge of Pretrained Language Models for Challenging Text Classification Settings
by: Gao, Lingyu
Published: (2024)

How Transferable are Attribute Controllers on Pretrained Multilingual Translation Models?
by: Liu, Danni, et al.
Published: (2023)

Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence
by: McLeish, Sean, et al.
Published: (2025)

BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Text
by: Bolton, Elliot, et al.
Published: (2024)

LoRA Meets Dropout under a Unified Framework
by: Wang, Sheng, et al.
Published: (2024)

Pretraining Large Language Models with NVFP4
by: NVIDIA, et al.
Published: (2025)

Patent Language Model Pretraining with ModernBERT
by: Yousefiramandi, Amirhossein, et al.
Published: (2025)