Image de couverture de livre

Morphological Typology in BPE Subword Productivity and Language Modeling

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteur principal:	Parra, Iñigo
Format:	Preprint
Publié:	2024
Sujets:	Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Pretraining Language Models with Subword Regularization: An Empirical Study of BPE Dropout in Low-Resource NLP
par: Visser, Ruan, et autres
Publié: (2026)

Evaluating Subword Tokenization Techniques for Bengali: A Benchmark Study with BengaliBPE
par: Patwary, Firoj Ahmmed, et autres
Publié: (2025)

The Learning Dynamics of Subword Segmentation for Morphologically Diverse Languages
par: Meyer, Francois, et autres
Publié: (2025)

Neural Correlates of Language Models Are Specific to Human Language
par: Parra, Iñigo
Publié: (2025)

UnMASKed: Quantifying Gender Biases in Masked Language Models through Linguistically Informed Job Market Prompts
par: Parra, Iñigo
Publié: (2024)

BlockBPE: Parallel BPE Tokenization
par: You, Amos
Publié: (2025)

Rethinking Tokenization for Rich Morphology: The Dominance of Unigram over BPE and Morphological Alignment
par: Vemula, Saketh Reddy, et autres
Publié: (2025)

SuperBPE: Space Travel for Language Models
par: Liu, Alisa, et autres
Publié: (2025)

Train It and Forget It: Merge Lists are Unnecessary for BPE Inference in Language Models
par: Sawada, Tomohiro, et autres
Publié: (2025)

Tokenization Falling Short: On Subword Robustness in Large Language Models
par: Chai, Yekun, et autres
Publié: (2024)

Adaptive BPE Tokenization for Enhanced Vocabulary Adaptation in Finetuning Pretrained Language Models
par: Balde, Gunjan, et autres
Publié: (2024)

Evaluating Morphological Plausibility of Subword Tokenization via Statistical Alignment with Morpho-Syntactic Features
par: Stephen, Abishek, et autres
Publié: (2026)

Understanding Subword Compositionality of Large Language Models
par: Peng, Qiwei, et autres
Publié: (2025)

IMPACT: Inflectional Morphology Probes Across Complex Typologies
par: Saeed, Mohammed J., et autres
Publié: (2025)

On the Effect of (Near) Duplicate Subwords in Language Modelling
par: Schäfer, Anton, et autres
Publié: (2024)

Batching BPE Tokenization Merges
par: Morgan, Alexander P.
Publié: (2024)

Evaluating Subword Tokenization: Alien Subword Composition and OOV Generalization Challenge
par: Batsuren, Khuyagbaatar, et autres
Publié: (2024)

Distributional Properties of Subword Regularization
par: Cognetta, Marco, et autres
Publié: (2024)

Lexically Grounded Subword Segmentation
par: Libovický, Jindřich, et autres
Publié: (2024)

Optimal Turkish Subword Strategies at Scale: Systematic Evaluation of Data, Vocabulary, Morphology Interplay
par: Altinok, Duygu
Publié: (2026)

Byte BPE Tokenization as an Inverse string Homomorphism
par: Geng, Saibo, et autres
Publié: (2024)

Bit-level BPE: Below the byte boundary
par: Moon, Sangwhan, et autres
Publié: (2025)

Decoupling the Benefits of Subword Tokenization for Language Model Training via Byte-level Simulation
par: Gigant, Théo, et autres
Publié: (2026)

Constructing a BPE Tokenization DFA
par: Berglund, Martin, et autres
Publié: (2024)

MoVoC: Morphology-Aware Subword Construction for Geez Script Languages
par: Teklehaymanot, Hailay Kidu, et autres
Publié: (2025)

Leading Whitespaces of Language Models' Subword Vocabulary Pose a Confound for Calculating Word Probabilities
par: Oh, Byung-Doh, et autres
Publié: (2024)

Can Pretrained Language Models Derive Correct Semantics from Corrupt Subwords under Noise?
par: Li, Xinzhe, et autres
Publié: (2023)

Stolen Subwords: Importance of Vocabularies for Machine Translation Model Stealing
par: Zouhar, Vilém
Publié: (2024)

An Analysis of BPE Vocabulary Trimming in Neural Machine Translation
par: Cognetta, Marco, et autres
Publié: (2024)

From Characters to Tokens: Dynamic Grouping with Hierarchical BPE
par: Dolga, Rares, et autres
Publié: (2025)

AdaptBPE: From General Purpose to Specialized Tokenizers
par: Liyanage, Vijini, et autres
Publié: (2026)

MorphBPE: A Morpho-Aware Tokenizer Bridging Linguistic Complexity for Efficient LLM Training Across Morphologies
par: Asgari, Ehsaneddin, et autres
Publié: (2025)

Scaffold-BPE: Enhancing Byte Pair Encoding for Large Language Models with Simple and Effective Scaffold Token Removal
par: Lian, Haoran, et autres
Publié: (2024)

Can Language Models Learn Typologically Implausible Languages?
par: Xu, Tianyang, et autres
Publié: (2025)

Tomato, Tomahto, Tomate: Do Multilingual Language Models Understand Based on Subword-Level Semantic Concepts?
par: Zhang, Crystina, et autres
Publié: (2024)

ByteSpan: Information-Driven Subword Tokenisation
par: Goriely, Zébulon, et autres
Publié: (2025)

Subword Tokenization Strategies for Kurdish Word Embeddings
par: Salehi, Ali, et autres
Publié: (2025)

BPE Stays on SCRIPT: Structured Encoding for Robust Multilingual Pretokenization
par: Land, Sander, et autres
Publié: (2025)

Differences in Typological Alignment in Language Models' Treatment of Differential Argument Marking
par: Deng, Iskar, et autres
Publié: (2026)

A Typologically Grounded Evaluation Framework for Word Order and Morphology Sensitivity in Multilingual Masked LMs
par: Feldman, Anna, et autres
Publié: (2026)