Cover Image

BPDec: Unveiling the Potential of Masked Language Modeling Decoder in BERT pretraining

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Liang, Wen, Liang, Youzhi
Format:	Preprint
Published:	2024
Subjects:	Computation and Language Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

MedicalBERT: enhancing biomedical natural language processing using pretrained BERT-based model
by: Reddy, K. Sahit, et al.
Published: (2025)

Chinese ModernBERT with Whole-Word Masking
by: Zhao, Zeyu, et al.
Published: (2025)

It's All in The [MASK]: Simple Instruction-Tuning Enables BERT-like Masked Language Models As Generative Classifiers
by: Clavié, Benjamin, et al.
Published: (2025)

Mask-guided BERT for Few Shot Text Classification
by: Liao, Wenxiong, et al.
Published: (2023)

Synthetic bootstrapped pretraining
by: Yang, Zitong, et al.
Published: (2025)

Empirical Analysis of Decoding Biases in Masked Diffusion Models
by: Huang, Pengcheng, et al.
Published: (2025)

StableMask: Refining Causal Masking in Decoder-only Transformer
by: Yin, Qingyu, et al.
Published: (2024)

Entropy-aware Masking for Masked Language Modeling
by: Srinivasagan, Gokul, et al.
Published: (2026)

Direct Alignment of Language Models via Quality-Aware Self-Refinement
by: Yu, Runsheng, et al.
Published: (2024)

Taming Masked Diffusion Language Models via Consistency Trajectory Reinforcement Learning with Fewer Decoding Step
by: Yang, Jingyi, et al.
Published: (2025)

Unveiling the Misuse Potential of Base Large Language Models via In-Context Learning
by: Wang, Xiao, et al.
Published: (2024)

Synthetic continued pretraining
by: Yang, Zitong, et al.
Published: (2024)

RooseBERT: A New Deal For Political Language Modelling
by: Dore, Deborah, et al.
Published: (2025)

KliniskVestBERT: BERT Model Specialised to Norwegian Clinical Texts
by: Autenried, Christian, et al.
Published: (2026)

Inconsistencies in Masked Language Models
by: Young, Tom, et al.
Published: (2022)

Large Language Models are Contrastive Reasoners
by: Yao, Liang
Published: (2024)

SED: Self-Evaluation Decoding Enhances Large Language Models for Better Generation
by: Luo, Ziqin, et al.
Published: (2024)

KuBERT: Central Kurdish BERT Model and Its Application for Sentiment Analysis
by: Awlla, Kozhin muhealddin, et al.
Published: (2025)

Unlocking the Potential: Benchmarking Large Language Models in Water Engineering and Research
by: Xu, Boyan, et al.
Published: (2024)

SoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compression
by: Huang, Xinhao, et al.
Published: (2026)

NeoBERT: A Next-Generation BERT
by: Breton, Lola Le, et al.
Published: (2025)

Beyond Hard Masks: Progressive Token Evolution for Diffusion Language Models
by: Zhong, Linhao, et al.
Published: (2026)

Patent Language Model Pretraining with ModernBERT
by: Yousefiramandi, Amirhossein, et al.
Published: (2025)

AraPoemBERT: A Pretrained Language Model for Arabic Poetry Analysis
by: Qarah, Faisal
Published: (2024)

SaudiBERT: A Large Language Model Pretrained on Saudi Dialect Corpora
by: Qarah, Faisal
Published: (2024)

Exploring Narrative Clustering in Large Language Models: A Layerwise Analysis of BERT
by: Banerjee, Awritrojit, et al.
Published: (2025)

Large Language Model as a Universal Clinical Multi-task Decoder
by: Wu, Yujiang, et al.
Published: (2024)

EuroBERT: Scaling Multilingual Encoders for European Languages
by: Boizard, Nicolas, et al.
Published: (2025)

Unveiling Factual Recall Behaviors of Large Language Models through Knowledge Neurons
by: Wang, Yifei, et al.
Published: (2024)

Between Circuits and Chomsky: Pre-pretraining on Formal Languages Imparts Linguistic Biases
by: Hu, Michael Y., et al.
Published: (2025)

Understanding the Interplay of Scale, Data, and Bias in Language Models: A Case Study with BERT
by: Ali, Muhammad, et al.
Published: (2024)

PhayaThaiBERT: Enhancing a Pretrained Thai Language Model with Unassimilated Loanwords
by: Sriwirote, Panyut, et al.
Published: (2023)

Selecting Between BERT and GPT for Text Classification in Political Science Research
by: Wang, Yu, et al.
Published: (2024)

Optimizing Decoding Paths in Masked Diffusion Models by Quantifying Uncertainty
by: Chen, Ziyu, et al.
Published: (2025)

Diffutron: A Masked Diffusion Language Model for Turkish Language
by: Kocabay, Şuayp Talha, et al.
Published: (2026)

Unitary Multi-Margin BERT for Robust Natural Language Processing
by: Chang, Hao-Yuan, et al.
Published: (2024)

Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding
by: Wang, Xintong, et al.
Published: (2024)

Parallelism and Generation Order in Masked Diffusion Language Models: Limits Today, Potential Tomorrow
by: Zhong, Yangyang, et al.
Published: (2026)

Analyzing Narrative Processing in Large Language Models (LLMs): Using GPT4 to test BERT
by: Krauss, Patrick, et al.
Published: (2024)

Relational Schemata in BERT Are Inducible, Not Emergent: A Study of Performance vs. Competence in Language Models
by: Gawin, Cole
Published: (2025)