Cover Image

Morphological Typology in BPE Subword Productivity and Language Modeling

Fuente: arXiv

Saved in:

Bibliographic Details
Main Author:	Parra, Iñigo
Format:	Preprint
Published:	2024
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Pretraining Language Models with Subword Regularization: An Empirical Study of BPE Dropout in Low-Resource NLP
by: Visser, Ruan, et al.
Published: (2026)

Evaluating Subword Tokenization Techniques for Bengali: A Benchmark Study with BengaliBPE
by: Patwary, Firoj Ahmmed, et al.
Published: (2025)

The Learning Dynamics of Subword Segmentation for Morphologically Diverse Languages
by: Meyer, Francois, et al.
Published: (2025)

Neural Correlates of Language Models Are Specific to Human Language
by: Parra, Iñigo
Published: (2025)

UnMASKed: Quantifying Gender Biases in Masked Language Models through Linguistically Informed Job Market Prompts
by: Parra, Iñigo
Published: (2024)

BlockBPE: Parallel BPE Tokenization
by: You, Amos
Published: (2025)

Rethinking Tokenization for Rich Morphology: The Dominance of Unigram over BPE and Morphological Alignment
by: Vemula, Saketh Reddy, et al.
Published: (2025)

SuperBPE: Space Travel for Language Models
by: Liu, Alisa, et al.
Published: (2025)

Train It and Forget It: Merge Lists are Unnecessary for BPE Inference in Language Models
by: Sawada, Tomohiro, et al.
Published: (2025)

Tokenization Falling Short: On Subword Robustness in Large Language Models
by: Chai, Yekun, et al.
Published: (2024)

Adaptive BPE Tokenization for Enhanced Vocabulary Adaptation in Finetuning Pretrained Language Models
by: Balde, Gunjan, et al.
Published: (2024)

Evaluating Morphological Plausibility of Subword Tokenization via Statistical Alignment with Morpho-Syntactic Features
by: Stephen, Abishek, et al.
Published: (2026)

Understanding Subword Compositionality of Large Language Models
by: Peng, Qiwei, et al.
Published: (2025)

IMPACT: Inflectional Morphology Probes Across Complex Typologies
by: Saeed, Mohammed J., et al.
Published: (2025)

On the Effect of (Near) Duplicate Subwords in Language Modelling
by: Schäfer, Anton, et al.
Published: (2024)

Batching BPE Tokenization Merges
by: Morgan, Alexander P.
Published: (2024)

Evaluating Subword Tokenization: Alien Subword Composition and OOV Generalization Challenge
by: Batsuren, Khuyagbaatar, et al.
Published: (2024)

Distributional Properties of Subword Regularization
by: Cognetta, Marco, et al.
Published: (2024)

Lexically Grounded Subword Segmentation
by: Libovický, Jindřich, et al.
Published: (2024)

Optimal Turkish Subword Strategies at Scale: Systematic Evaluation of Data, Vocabulary, Morphology Interplay
by: Altinok, Duygu
Published: (2026)

Byte BPE Tokenization as an Inverse string Homomorphism
by: Geng, Saibo, et al.
Published: (2024)

Bit-level BPE: Below the byte boundary
by: Moon, Sangwhan, et al.
Published: (2025)

Decoupling the Benefits of Subword Tokenization for Language Model Training via Byte-level Simulation
by: Gigant, Théo, et al.
Published: (2026)

Constructing a BPE Tokenization DFA
by: Berglund, Martin, et al.
Published: (2024)

MoVoC: Morphology-Aware Subword Construction for Geez Script Languages
by: Teklehaymanot, Hailay Kidu, et al.
Published: (2025)

Leading Whitespaces of Language Models' Subword Vocabulary Pose a Confound for Calculating Word Probabilities
by: Oh, Byung-Doh, et al.
Published: (2024)

Can Pretrained Language Models Derive Correct Semantics from Corrupt Subwords under Noise?
by: Li, Xinzhe, et al.
Published: (2023)

Stolen Subwords: Importance of Vocabularies for Machine Translation Model Stealing
by: Zouhar, Vilém
Published: (2024)

An Analysis of BPE Vocabulary Trimming in Neural Machine Translation
by: Cognetta, Marco, et al.
Published: (2024)

From Characters to Tokens: Dynamic Grouping with Hierarchical BPE
by: Dolga, Rares, et al.
Published: (2025)

AdaptBPE: From General Purpose to Specialized Tokenizers
by: Liyanage, Vijini, et al.
Published: (2026)

MorphBPE: A Morpho-Aware Tokenizer Bridging Linguistic Complexity for Efficient LLM Training Across Morphologies
by: Asgari, Ehsaneddin, et al.
Published: (2025)

Scaffold-BPE: Enhancing Byte Pair Encoding for Large Language Models with Simple and Effective Scaffold Token Removal
by: Lian, Haoran, et al.
Published: (2024)

Can Language Models Learn Typologically Implausible Languages?
by: Xu, Tianyang, et al.
Published: (2025)

Tomato, Tomahto, Tomate: Do Multilingual Language Models Understand Based on Subword-Level Semantic Concepts?
by: Zhang, Crystina, et al.
Published: (2024)

ByteSpan: Information-Driven Subword Tokenisation
by: Goriely, Zébulon, et al.
Published: (2025)

Subword Tokenization Strategies for Kurdish Word Embeddings
by: Salehi, Ali, et al.
Published: (2025)

BPE Stays on SCRIPT: Structured Encoding for Robust Multilingual Pretokenization
by: Land, Sander, et al.
Published: (2025)

Differences in Typological Alignment in Language Models' Treatment of Differential Argument Marking
by: Deng, Iskar, et al.
Published: (2026)

A Typologically Grounded Evaluation Framework for Word Order and Morphology Sensitivity in Multilingual Masked LMs
by: Feldman, Anna, et al.
Published: (2026)