Image de couverture de livre

Spelling-out is not Straightforward: LLMs' Capability of Tokenization from Token to Characters

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Hiraoka, Tatsuya, Inui, Kentaro
Format:	Preprint
Publié:	2025
Sujets:	Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Repetition Neurons: How Do Language Models Produce Repetitions?
par: Hiraoka, Tatsuya, et autres
Publié: (2024)

Understanding and Controlling Repetition Neurons and Induction Heads in In-Context Learning
par: Doan, Nhi Hoai, et autres
Publié: (2025)

Knowledge of Pretrained Language Models on Surface Information of Tokens
par: Hiraoka, Tatsuya, et autres
Publié: (2024)

Tokenization Preference for Human and Machine Learning Model: An Annotation Study
par: Hiraoka, Tatsuya, et autres
Publié: (2023)

The Geometry of Numerical Reasoning: Language Models Compare Numeric Properties in Linear Subspaces
par: El-Shangiti, Ahmed Oumar, et autres
Publié: (2024)

Sycophancy Hides Linearly in the Attention Heads
par: Genadi, Rifo, et autres
Publié: (2026)

Number Representations in LLMs: A Computational Parallel to Human Perception
par: AlquBoj, H. V., et autres
Publié: (2025)

Rectifying Belief Space via Unlearning to Harness LLMs' Reasoning
par: Niwa, Ayana, et autres
Publié: (2025)

Do LLMs Need to Think in One Language? Correlation between Latent Language and Task Performance
par: Ozaki, Shintaro, et autres
Publié: (2025)

Enhancing Character-Level Understanding in LLMs through Token Internal Structure Learning
par: Xu, Zhu, et autres
Publié: (2024)

C-LLM: Learn to Check Chinese Spelling Errors Character by Character
par: Li, Kunting, et autres
Publié: (2024)

Monotonic Representation of Numeric Properties in Language Models
par: Heinzerling, Benjamin, et autres
Publié: (2024)

From Characters to Tokens: Dynamic Grouping with Hierarchical BPE
par: Dolga, Rares, et autres
Publié: (2025)

RECALL: Library-Like Behavior In Language Models is Enhanced by Self-Referencing Causal Cycles
par: Nwadike, Munachiso, et autres
Publié: (2025)

CharBench: Evaluating the Role of Tokenization in Character-Level Tasks
par: Uzan, Omri, et autres
Publié: (2025)

Broken-Token: Filtering Obfuscated Prompts by Counting Characters-Per-Token
par: Zychlinski, Shaked, et autres
Publié: (2025)

Token Alignment via Character Matching for Subword Completion
par: Athiwaratkun, Ben, et autres
Publié: (2024)

Behavior-Equivalent Token: Single-Token Replacement for Long Prompts in LLMs
par: Dong, Jiancheng, et autres
Publié: (2025)

Investigating Neurons and Heads in Transformer-based LLMs for Typographical Errors
par: Tsuji, Kohei, et autres
Publié: (2025)

The Strawberry Problem: Emergence of Character-level Understanding in Tokenized Language Models
par: Cosma, Adrian, et autres
Publié: (2025)

Cell-Based Representation of Relational Binding in Language Models
par: Dai, Qin, et autres
Publié: (2026)

Representational Analysis of Binding in Language Models
par: Dai, Qin, et autres
Publié: (2024)

Empowering Character-level Text Infilling by Eliminating Sub-Tokens
par: Ren, Houxing, et autres
Publié: (2024)

Bit-level BPE: Below the byte boundary
par: Moon, Sangwhan, et autres
Publié: (2025)

CUTE: Measuring LLMs' Understanding of Their Tokens
par: Edman, Lukas, et autres
Publié: (2024)

Word Recovery in Large Language Models Enables Character-Level Tokenization Robustness
par: Yang, Zhipeng, et autres
Publié: (2026)

KL3M Tokenizers: A Family of Domain-Specific and Character-Level Tokenizers for Legal, Financial, and Preprocessing Applications
par: Bommarito, Michael J, et autres
Publié: (2025)

From Language Models over Tokens to Language Models over Characters
par: Vieira, Tim, et autres
Publié: (2024)

LLMs Can Compensate for Deficiencies in Visual Representations
par: Takishita, Sho, et autres
Publié: (2025)

Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capability
par: Lin, Zicheng, et autres
Publié: (2024)

StableToken: A Noise-Robust Semantic Speech Tokenizer for Resilient SpeechLLMs
par: Song, Yuhan, et autres
Publié: (2025)

DISC: Plug-and-Play Decoding Intervention with Similarity of Characters for Chinese Spelling Check
par: Qiao, Ziheng, et autres
Publié: (2024)

Benchmarking Prosody Encoding in Discrete Speech Tokens
par: Onda, Kentaro, et autres
Publié: (2025)

Toward a Theory of Tokenization in LLMs
par: Rajaraman, Nived, et autres
Publié: (2024)

LLMs are Not Just Next Token Predictors
par: Downes, Stephen M., et autres
Publié: (2024)

Speculating LLMs' Chinese Training Data Pollution from Their Tokens
par: Zhang, Qingjie, et autres
Publié: (2025)

Optimizing Korean-Centric LLMs via Token Pruning
par: Kim, Hoyeol, et autres
Publié: (2026)

Beyond Tokens: Concept-Level Training Objectives for LLMs
par: Iyer, Laya, et autres
Publié: (2026)

Accelerating Production LLMs with Combined Token/Embedding Speculators
par: Wertheimer, Davis, et autres
Publié: (2024)

FastSpell: the LangId Magic Spell
par: Bañón, Marta, et autres
Publié: (2024)