Cover Image

Pretraining Language Models with Subword Regularization: An Empirical Study of BPE Dropout in Low-Resource NLP

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Visser, Ruan, Grobler, Trienko, Dunaiski, Marcel
Format:	Preprint
Published:	2026
Subjects:	Computation and Language Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Combining Language and Topic Models for Hierarchical Text Classification
by: Toit, Jaco du, et al.
Published: (2025)

SuperBPE: Space Travel for Language Models
by: Liu, Alisa, et al.
Published: (2025)

Morphological Typology in BPE Subword Productivity and Language Modeling
by: Parra, Iñigo
Published: (2024)

Understanding Subword Compositionality of Large Language Models
by: Peng, Qiwei, et al.
Published: (2025)

On Importance of Pruning and Distillation for Efficient Low Resource NLP
by: Mirashi, Aishwarya, et al.
Published: (2024)

Generalizable and Stable Finetuning of Pretrained Language Models on Low-Resource Texts
by: Somayajula, Sai Ashish, et al.
Published: (2024)

On the Effect of (Near) Duplicate Subwords in Language Modelling
by: Schäfer, Anton, et al.
Published: (2024)

Trans-Tokenization and Cross-lingual Vocabulary Transfers: Language Adaptation of LLMs for Low-Resource NLP
by: Remy, François, et al.
Published: (2024)

Towards Efficient Active Learning in NLP via Pretrained Representations
by: Vysogorets, Artem, et al.
Published: (2024)

Constructing a BPE Tokenization DFA
by: Berglund, Martin, et al.
Published: (2024)

AF Adapter: Continual Pretraining for Building Chinese Biomedical Language Model
by: Yan, Yongyu, et al.
Published: (2022)

DPad: Efficient Diffusion Language Models with Suffix Dropout
by: Chen, Xinhua, et al.
Published: (2025)

Does Differential Privacy Impact Bias in Pretrained NLP Models?
by: Islam, Md. Khairul, et al.
Published: (2024)

Evaluating Subword Tokenization Techniques for Bengali: A Benchmark Study with BengaliBPE
by: Patwary, Firoj Ahmmed, et al.
Published: (2025)

Learning Mutually Informed Representations for Characters and Subwords
by: Wang, Yilin, et al.
Published: (2023)

Investigating the Synergistic Effects of Dropout and Residual Connections on Language Model Training
by: Li, Qingyang, et al.
Published: (2024)

Large Language Models for Math Education in Low-Resource Languages: A Study in Sinhala and Tamil
by: Kishanthan, Sukumar, et al.
Published: (2026)

An Empirical Study of Mamba-based Language Models
by: Waleffe, Roger, et al.
Published: (2024)

Data Mixture Inference: What do BPE Tokenizers Reveal about their Training Data?
by: Hayase, Jonathan, et al.
Published: (2024)

Textless NLP -- Zero Resource Challenge with Low Resource Compute
by: Ramadass, Krithiga, et al.
Published: (2024)

Aligning Large Language Models to Low-Resource Languages through LLM-Based Selective Translation: A Systematic Study
by: Paul, Rakesh, et al.
Published: (2025)

Assessing Adversarial Robustness of Large Language Models: An Empirical Study
by: Yang, Zeyu, et al.
Published: (2024)

Factual Consistency of Multilingual Pretrained Language Models
by: Fierro, Constanza, et al.
Published: (2022)

Not All Pretraining are Created Equal: Threshold Tuning and Class Weighting for Imbalanced Polarization Tasks in Low-Resource Settings
by: Oguntade, Abass
Published: (2026)

On Importance of Layer Pruning for Smaller BERT Models and Low Resource Languages
by: Shirke, Mayur, et al.
Published: (2025)

SampoNLP: A Self-Referential Toolkit for Morphological Analysis of Subword Tokenizers
by: Chelombitko, Iaroslav, et al.
Published: (2026)

On Limitations of LLM as Annotator for Low Resource Languages
by: Jadhav, Suramya, et al.
Published: (2024)

Unsupervised Pretraining for Fact Verification by Language Model Distillation
by: Bazaga, Adrián, et al.
Published: (2023)

NLP-ADBench: NLP Anomaly Detection Benchmark
by: Li, Yuangang, et al.
Published: (2024)

Enhancing Sindhi Word Segmentation using Subword Representation Learning and Position-aware Self-attention
by: Ali, Wazir, et al.
Published: (2020)

LakotaBERT: A Transformer-based Model for Low Resource Lakota Language
by: Parankusham, Kanishka, et al.
Published: (2025)

Privacy Evaluation Benchmarks for NLP Models
by: Huang, Wei, et al.
Published: (2024)

ProxSparse: Regularized Learning of Semi-Structured Sparsity Masks for Pretrained LLMs
by: Liu, Hongyi, et al.
Published: (2025)

An Empirical Study of Multi-Generation Sampling for Jailbreak Detection in Large Language Models
by: Luo, Hanrui, et al.
Published: (2026)

Unveiling Factors for Enhanced POS Tagging: A Study of Low-Resource Medieval Romance Languages
by: Schöffel, Matthias, et al.
Published: (2025)

Basis Selection: Low-Rank Decomposition of Pretrained Large Language Models for Target Applications
by: Li, Yang, et al.
Published: (2024)

Kakugo: Distillation of Low-Resource Languages into Small Language Models
by: Devine, Peter, et al.
Published: (2026)

Procedural Pretraining: Warming Up Language Models with Abstract Data
by: Jiang, Liangze, et al.
Published: (2026)

GPUTOK: GPU Accelerated Byte Level BPE Tokenization
by: Kadamba, Venu Gopal, et al.
Published: (2026)

Facts in Stats: Impacts of Pretraining Diversity on Language Model Generalization
by: Behnia, Tina, et al.
Published: (2025)