Cover Image

Dynamic Masking Rate Schedules for MLM Pretraining

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Ankner, Zachary, Saphra, Naomi, Blalock, Davis, Frankle, Jonathan, Leavitt, Matthew L.
Format:	Preprint
Published:	2023
Subjects:	Computation and Language Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Mechanistic?
by: Saphra, Naomi, et al.
Published: (2024)

ChatGPT Doesn't Trust Chargers Fans: Guardrail Sensitivity in Context
by: Li, Victoria R., et al.
Published: (2024)

Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMs
by: Chen, Angelica, et al.
Published: (2023)

Can Interpretation Predict Behavior on Unseen Data?
by: Li, Victoria R., et al.
Published: (2025)

Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models
by: Ankner, Zachary, et al.
Published: (2024)

Power Scheduler: A Batch Size and Token Number Agnostic Learning Rate Scheduler
by: Shen, Yikang, et al.
Published: (2024)

MaskTab: Scalable Masked Tabular Pretraining with Scaling Laws and Distillation for Industrial Classification
by: Zheng, Bo, et al.
Published: (2026)

Using Shapley interactions to understand how models use structure
by: Singhvi, Divyansh, et al.
Published: (2024)

Rethinking the Role of Text Complexity in Language Model Pretraining
by: Velasco, Dan John, et al.
Published: (2025)

Learning Dynamics of Meta-Learning in Small Model Pretraining
by: Africa, David Demitri, et al.
Published: (2025)

Beyond Repetition: Text Simplification and Curriculum Learning for Data-Constrained Pretraining
by: Roque, Matthew Theodore, et al.
Published: (2025)

Scaling, Simplification, and Adaptation: Lessons from Pretraining on Machine-Translated Text
by: Velasco, Dan John, et al.
Published: (2025)

Unsupervised Elicitation of Language Models
by: Wen, Jiaxin, et al.
Published: (2025)

BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Text
by: Bolton, Elliot, et al.
Published: (2024)

Memorization Dynamics of Fill-in-the-Middle Pretraining
by: von Arx, Tobias, et al.
Published: (2026)

Automated Conversion of Static to Dynamic Scheduler via Natural Language
by: Tang, Paul Mingzheng, et al.
Published: (2024)

Recite, Reconstruct, Recollect: Memorization in LMs as a Multifaceted Phenomenon
by: Prashanth, USVSN Sai, et al.
Published: (2024)

Entropy-aware Masking for Masked Language Modeling
by: Srinivasagan, Gokul, et al.
Published: (2026)

D$^3$: Dynamic Directional Graph-Constrained Data Scheduling for LLM Training
by: Xu, Yuanjian, et al.
Published: (2026)

Trainable Dynamic Mask Sparse Attention
by: Shi, Jingze, et al.
Published: (2025)

StableMask: Refining Causal Masking in Decoder-only Transformer
by: Yin, Qingyu, et al.
Published: (2024)

DAM: Dynamic Attention Mask for Long-Context Large Language Model Inference Acceleration
by: Zhang, Hanzhi, et al.
Published: (2025)

How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining
by: Luo, Kairong, et al.
Published: (2025)

Intermittent Semi-Working Mask: A New Masking Paradigm for LLMs
by: Hu, HaoYuan, et al.
Published: (2024)

Many-to-English Machine Translation Tools, Data, and Pretrained Models
by: Gowda, Thamme, et al.
Published: (2021)

Multilingual Pretraining for Pixel Language Models
by: Kesen, Ilker, et al.
Published: (2025)

A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedules
by: Luo, Kairong, et al.
Published: (2025)

MuRating: A High Quality Data Selecting Approach to Multilingual Large Language Model Pretraining
by: Chen, Zhixun, et al.
Published: (2025)

WRAP++: Web discoveRy Amplified Pretraining
by: Zhou, Jiang, et al.
Published: (2026)

Tracing Persona Vectors Through LLM Pretraining
by: Moskvoretskii, Viktor, et al.
Published: (2026)

Attributing Culture-Conditioned Generations to Pretraining Corpora
by: Li, Huihan, et al.
Published: (2024)

Pretrained LLMs Learn Multiple Types of Uncertainty
by: Cohen, Roi, et al.
Published: (2025)

daVinci-LLM:Towards the Science of Pretraining
by: Qin, Yiwei, et al.
Published: (2026)

Inconsistencies in Masked Language Models
by: Young, Tom, et al.
Published: (2022)

On Linear Representations and Pretraining Data Frequency in Language Models
by: Merullo, Jack, et al.
Published: (2025)

Drop Dropout on Single-Epoch Language Model Pretraining
by: Liu, Houjun, et al.
Published: (2025)

Is It a Free Lunch for Removing Outliers during Pretraining?
by: Liao, Baohao, et al.
Published: (2024)

AI Managed Emergency Documentation with a Pretrained Model
by: Menzies, David, et al.
Published: (2024)

Optimizing Pretraining Data Mixtures with LLM-Estimated Utility
by: Held, William, et al.
Published: (2025)

Gender Encoding Patterns in Pretrained Language Model Representations
by: Zakizadeh, Mahdi, et al.
Published: (2025)