:: Library Catalog

Cover Image

Saved in:

Bibliographic Details
Main Authors:	Salehi, Ali, Jacobs, Cassandra L.
Format:	Preprint
Published:	2025
Subjects:	Computation and Language
Online Access:	https://arxiv.org/abs/2511.14696
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Evaluating Subword Tokenization: Alien Subword Composition and OOV Generalization Challenge
by: Batsuren, Khuyagbaatar, et al.
Published: (2024)

Tokenization Falling Short: On Subword Robustness in Large Language Models
by: Chai, Yekun, et al.
Published: (2024)

On the scaling relationship between cloze probabilities and language model next-token prediction
by: Jacobs, Cassandra L., et al.
Published: (2026)

Statistical Uncertainty in Word Embeddings: GloVe-V
by: Vallebueno, Andrea, et al.
Published: (2024)

Token Alignment via Character Matching for Subword Completion
by: Athiwaratkun, Ben, et al.
Published: (2024)

Enhancing Sindhi Word Segmentation using Subword Representation Learning and Position-aware Self-attention
by: Ali, Wazir, et al.
Published: (2020)

T-FREE: Subword Tokenizer-Free Generative LLMs via Sparse Representations for Memory-Efficient Embeddings
by: Deiseroth, Björn, et al.
Published: (2024)

Assessing the Importance of Frequency versus Compositionality for Subword-based Tokenization in NMT
by: Wolleb, Benoist, et al.
Published: (2023)

The distribution of discourse relations within and across turns in spontaneous conversation
by: Cortez, S. Magalí López, et al.
Published: (2023)

A Bayesian account of pronoun and neopronoun acquisition
by: Jacobs, Cassandra L., et al.
Published: (2025)

Evaluating Subword Tokenization Techniques for Bengali: A Benchmark Study with BengaliBPE
by: Patwary, Firoj Ahmmed, et al.
Published: (2025)

A Subword Embedding Approach for Variation Detection in Luxembourgish User Comments
by: Lutgen, Anne-Marie, et al.
Published: (2026)

Team Ryu's Submission to SIGMORPHON 2024 Shared Task on Subword Tokenization
by: Li, Zilong
Published: (2024)

Leading Whitespaces of Language Models' Subword Vocabulary Pose a Confound for Calculating Word Probabilities
by: Oh, Byung-Doh, et al.
Published: (2024)

Evaluating Morphological Plausibility of Subword Tokenization via Statistical Alignment with Morpho-Syntactic Features
by: Stephen, Abishek, et al.
Published: (2026)

Decoupling the Benefits of Subword Tokenization for Language Model Training via Byte-level Simulation
by: Gigant, Théo, et al.
Published: (2026)

Distributional Properties of Subword Regularization
by: Cognetta, Marco, et al.
Published: (2024)

Lexically Grounded Subword Segmentation
by: Libovický, Jindřich, et al.
Published: (2024)

Linguistic Laws Meet Protein Sequences: A Comparative Analysis of Subword Tokenization Methods
by: Suyunu, Burak, et al.
Published: (2024)

LGSE: Lexically Grounded Subword Embedding Initialization for Low-Resource Language Adaptation
by: Teklehaymanot, Hailay, et al.
Published: (2026)

Idiom Detection in Sorani Kurdish Texts
by: Omer, Skala Kamaran, et al.
Published: (2025)

OFA: A Framework of Initializing Unseen Subword Embeddings for Efficient Large-scale Multilingual Continued Pretraining
by: Liu, Yihong, et al.
Published: (2023)

Large-scale cloze evaluation reveals that token prediction tasks are neither lexically nor semantically aligned
by: Jacobs, Cassandra L., et al.
Published: (2024)

ByteSpan: Information-Driven Subword Tokenisation
by: Goriely, Zébulon, et al.
Published: (2025)

Language and Speech Technology for Central Kurdish Varieties
by: Ahmadi, Sina, et al.
Published: (2024)

Optimal Turkish Subword Strategies at Scale: Systematic Evaluation of Data, Vocabulary, Morphology Interplay
by: Altinok, Duygu
Published: (2026)

An Evaluation of Sindhi Word Embedding in Semantic Analogies and Downstream Tasks
by: Ali, Wazir, et al.
Published: (2024)

Subword models struggle with word learning, but surprisal hides it
by: Bunzeck, Bastian, et al.
Published: (2025)

The Learning Dynamics of Subword Segmentation for Morphologically Diverse Languages
by: Meyer, Francois, et al.
Published: (2025)

Morphological Typology in BPE Subword Productivity and Language Modeling
by: Parra, Iñigo
Published: (2024)

FLEURS-Kobani: Extending the FLEURS Dataset for Northern Kurdish
by: Jaff, Daban Q., et al.
Published: (2026)

Learning Mutually Informed Representations for Characters and Subwords
by: Wang, Yilin, et al.
Published: (2023)

KurdSTS: The Kurdish Semantic Textual Similarity
by: Abdullah, Abdulhady Abas, et al.
Published: (2025)

One Word Is Not Enough: Simple Prompts Improve Word Embeddings
by: Ranjan, Rajeev
Published: (2025)

Less is more: Probabilistic reduction is best explained by small-scale predictability measures
by: Jacobs, Cassandra L., et al.
Published: (2025)

StochasTok: Improving Fine-Grained Subword Understanding in LLMs
by: Sims, Anya, et al.
Published: (2025)

Stolen Subwords: Importance of Vocabularies for Machine Translation Model Stealing
by: Zouhar, Vilém
Published: (2024)

Automatic Text Summarization (ATS) for Research Documents in Sorani Kurdish
by: Abdulrahman, Rondik Hadi, et al.
Published: (2025)

Tokenization Strategies for Low-Resource Agglutinative Languages in Word2Vec: Case Study on Turkish and Finnish
by: Hu, Jinfan Frank
Published: (2025)

Existential Definability over the Subword Ordering
by: Baumann, Pascal, et al.
Published: (2022)