Cover Image

Broken Tokens? Your Language Model can Secretly Handle Non-Canonical Tokenizations

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Zheng, Brian Siyuan, Liu, Alisa, Ahia, Orevaoghene, Hayase, Jonathan, Choi, Yejin, Smith, Noah A.
Format:	Preprint
Published:	2025
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Are you going to finish that? A Practical Study of the Partial Token Problem
by: Xu, Hao, et al.
Published: (2026)

Data Mixture Inference: What do BPE Tokenizers Reveal about their Training Data?
by: Hayase, Jonathan, et al.
Published: (2024)

FLEXITOKENS: Flexible Tokenization for Evolving Language Models
by: Owodunni, Abraham Toluwase, et al.
Published: (2025)

Sampling from Your Language Model One Byte at a Time
by: Hayase, Jonathan, et al.
Published: (2025)

MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenization
by: Ahia, Orevaoghene, et al.
Published: (2024)

SuperBPE: Space Travel for Language Models
by: Liu, Alisa, et al.
Published: (2025)

Tuning Language Models by Proxy
by: Liu, Alisa, et al.
Published: (2024)

BASS: Benchmarking Audio LMs for Musical Structure and Semantic Reasoning
by: Jang, Min, et al.
Published: (2026)

MYTE: Morphology-Driven Byte Encoding for Better and Fairer Multilingual Language Modeling
by: Limisiewicz, Tomasz, et al.
Published: (2024)

Extracting Lexical Features from Dialects via Interpretable Dialect Classifiers
by: Xie, Roy, et al.
Published: (2024)

Voices Unheard: NLP Resources and Models for Yorùbá Regional Dialects
by: Ahia, Orevaoghene, et al.
Published: (2024)

Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens
by: Liu, Jiacheng, et al.
Published: (2024)

Broken Words, Broken Performance: Effect of Tokenization on Performance of LLMs
by: Pawar, Sachin, et al.
Published: (2025)

Frame-Level Internal Tool Use for Temporal Grounding in Audio LMs
by: An, Joesph, et al.
Published: (2026)

What Language is This? Ask Your Tokenizer
by: Meister, Clara, et al.
Published: (2026)

DIALECTBENCH: A NLP Benchmark for Dialects, Varieties, and Closely-Related Languages
by: Faisal, Fahim, et al.
Published: (2024)

Broken-Token: Filtering Obfuscated Prompts by Counting Characters-Per-Token
by: Zychlinski, Shaked, et al.
Published: (2025)

Critical Learning Periods: Leveraging Early Training Dynamics for Efficient Data Pruning
by: Chimoto, Everlyn Asiko, et al.
Published: (2024)

Symbolic Working Memory Enhances Language Models for Complex Rule Application
by: Wang, Siyuan, et al.
Published: (2024)

Compute Optimal Tokenization
by: Limisiewicz, Tomasz, et al.
Published: (2026)

Does Liking Yellow Imply Driving a School Bus? Semantic Leakage in Language Models
by: Gonen, Hila, et al.
Published: (2024)

OLMoTrace: Tracing Language Model Outputs Back to Trillions of Training Tokens
by: Liu, Jiacheng, et al.
Published: (2025)

Problematic Tokens: Tokenizer Bias in Large Language Models
by: Yang, Jin, et al.
Published: (2024)

Teaching LLMs to Abstain across Languages via Multilingual Feedback
by: Feng, Shangbin, et al.
Published: (2024)

From Next-Token to Mathematics: The Learning Dynamics of Mathematical Reasoning in Language Models
by: Mishra, Shubhra, et al.
Published: (2024)

Infini-gram mini: Exact n-gram Search at the Internet Scale with FM-Index
by: Xu, Hao, et al.
Published: (2025)

Rethinking Tokenization: Crafting Better Tokenizers for Large Language Models
by: Yang, Jinbiao
Published: (2024)

FTP: A Fine-grained Token-wise Pruner for Large Language Models via Token Routing
by: Li, Zekai, et al.
Published: (2024)

IDIOLEX: Unified and Continuous Representations for Idiolectal and Stylistic Variation
by: Kantharuban, Anjali, et al.
Published: (2026)

SemToken: Semantic-Aware Tokenization for Efficient Long-Context Language Modeling
by: Liu, Dong, et al.
Published: (2025)

Learning to Compress Prompts with Gist Tokens
by: Mu, Jesse, et al.
Published: (2023)

Flexibly Scaling Large Language Models Contexts Through Extensible Tokenization
by: Shao, Ninglu, et al.
Published: (2024)

Disentangling Reasoning Tokens and Boilerplate Tokens For Language Model Fine-tuning
by: Ye, Ziang, et al.
Published: (2024)

One Token Is Enough: Improving Diffusion Language Models with a Sink Token
by: Zhang, Zihou, et al.
Published: (2026)

A Text is Worth Several Tokens: Text Embedding from LLMs Secretly Aligns Well with The Key Tokens
by: Nie, Zhijie, et al.
Published: (2024)

Your Language Model Secretly Contains Personality Subnetworks
by: Ye, Ruimeng, et al.
Published: (2026)

IPO: Your Language Model is Secretly a Preference Classifier
by: Garg, Shivank, et al.
Published: (2025)

Structured Token Retention and Computational Memory Paths in Large Language Models
by: Delena, Jonathan, et al.
Published: (2025)

LargePiG: Your Large Language Model is Secretly a Pointer Generator
by: Sun, Zhongxiang, et al.
Published: (2024)

Thinking Tokens for Language Modeling
by: Herel, David, et al.
Published: (2024)