Imagen de Portada

The Learning Dynamics of Subword Segmentation for Morphologically Diverse Languages

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Meyer, Francois, Buys, Jan
Formato:	Preprint
Publicado:	2025
Materias:	Computation and Language
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

A Systematic Analysis of Subwords and Cross-Lingual Transfer in Multilingual Translation
por: Meyer, Francois, et al.
Publicado: (2024)

Triples-to-isiXhosa (T2X): Addressing the Challenges of Low-Resource Agglutinative Data-to-Text Generation
por: Meyer, Francois, et al.
Publicado: (2024)

Morphological Typology in BPE Subword Productivity and Language Modeling
por: Parra, Iñigo
Publicado: (2024)

Lexically Grounded Subword Segmentation
por: Libovický, Jindřich, et al.
Publicado: (2024)

MzansiText and MzansiLM: An Open Corpus and Decoder-Only Language Model for South African Languages
por: Lombard, Anri, et al.
Publicado: (2026)

Neural Morphological Tagging for Nguni Languages
por: Marquard, Cael, et al.
Publicado: (2025)

Evaluating Morphological Plausibility of Subword Tokenization via Statistical Alignment with Morpho-Syntactic Features
por: Stephen, Abishek, et al.
Publicado: (2026)

Evaluating Subword Tokenization: Alien Subword Composition and OOV Generalization Challenge
por: Batsuren, Khuyagbaatar, et al.
Publicado: (2024)

Distributional Properties of Subword Regularization
por: Cognetta, Marco, et al.
Publicado: (2024)

Tokenization Falling Short: On Subword Robustness in Large Language Models
por: Chai, Yekun, et al.
Publicado: (2024)

Optimal Turkish Subword Strategies at Scale: Systematic Evaluation of Data, Vocabulary, Morphology Interplay
por: Altinok, Duygu
Publicado: (2026)

Enhancing Sindhi Word Segmentation using Subword Representation Learning and Position-aware Self-attention
por: Ali, Wazir, et al.
Publicado: (2020)

Learning Mutually Informed Representations for Characters and Subwords
por: Wang, Yilin, et al.
Publicado: (2023)

MoVoC: Morphology-Aware Subword Construction for Geez Script Languages
por: Teklehaymanot, Hailay Kidu, et al.
Publicado: (2025)

Understanding Subword Compositionality of Large Language Models
por: Peng, Qiwei, et al.
Publicado: (2025)

On the Effect of (Near) Duplicate Subwords in Language Modelling
por: Schäfer, Anton, et al.
Publicado: (2024)

ByteSpan: Information-Driven Subword Tokenisation
por: Goriely, Zébulon, et al.
Publicado: (2025)

Subword Tokenization Strategies for Kurdish Word Embeddings
por: Salehi, Ali, et al.
Publicado: (2025)

Decoupling the Benefits of Subword Tokenization for Language Model Training via Byte-level Simulation
por: Gigant, Théo, et al.
Publicado: (2026)

Subword models struggle with word learning, but surprisal hides it
por: Bunzeck, Bastian, et al.
Publicado: (2025)

Designing and Contextualising Probes for African Languages
por: Aduah, Wisdom, et al.
Publicado: (2025)

Leading Whitespaces of Language Models' Subword Vocabulary Pose a Confound for Calculating Word Probabilities
por: Oh, Byung-Doh, et al.
Publicado: (2024)

Can Pretrained Language Models Derive Correct Semantics from Corrupt Subwords under Noise?
por: Li, Xinzhe, et al.
Publicado: (2023)

Multipath parsing in the brain
por: Franzluebbers, Berta, et al.
Publicado: (2024)

LGSE: Lexically Grounded Subword Embedding Initialization for Low-Resource Language Adaptation
por: Teklehaymanot, Hailay, et al.
Publicado: (2026)

StochasTok: Improving Fine-Grained Subword Understanding in LLMs
por: Sims, Anya, et al.
Publicado: (2025)

Stolen Subwords: Importance of Vocabularies for Machine Translation Model Stealing
por: Zouhar, Vilém
Publicado: (2024)

Tomato, Tomahto, Tomate: Do Multilingual Language Models Understand Based on Subword-Level Semantic Concepts?
por: Zhang, Crystina, et al.
Publicado: (2024)

SubRegWeigh: Effective and Efficient Annotation Weighing with Subword Regularization
por: Tsuji, Kohei, et al.
Publicado: (2024)

A Subword Embedding Approach for Variation Detection in Luxembourgish User Comments
por: Lutgen, Anne-Marie, et al.
Publicado: (2026)

Assessing the Importance of Frequency versus Compositionality for Subword-based Tokenization in NMT
por: Wolleb, Benoist, et al.
Publicado: (2023)

Token Alignment via Character Matching for Subword Completion
por: Athiwaratkun, Ben, et al.
Publicado: (2024)

Existential Definability over the Subword Ordering
por: Baumann, Pascal, et al.
Publicado: (2022)

Pretraining Language Models with Subword Regularization: An Empirical Study of BPE Dropout in Low-Resource NLP
por: Visser, Ruan, et al.
Publicado: (2026)

Transferring Extreme Subword Style Using Ngram Model-Based Logit Scaling
por: Messner, Craig, et al.
Publicado: (2025)

Evaluating Subword Tokenization Techniques for Bengali: A Benchmark Study with BengaliBPE
por: Patwary, Firoj Ahmmed, et al.
Publicado: (2025)

SampoNLP: A Self-Referential Toolkit for Morphological Analysis of Subword Tokenizers
por: Chelombitko, Iaroslav, et al.
Publicado: (2026)

Team Ryu's Submission to SIGMORPHON 2024 Shared Task on Subword Tokenization
por: Li, Zilong
Publicado: (2024)

From Smør-re-brød to Subwords: Training LLMs on Danish, One Morpheme at a Time
por: Kildeberg, Mikkel Wildner, et al.
Publicado: (2025)

TAMS: Translation-Assisted Morphological Segmentation
por: Rice, Enora, et al.
Publicado: (2024)