Image de couverture de livre

LiteToken: Removing Intermediate Merge Residues From BPE Tokenizers

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Sun, Yike, Yang, Haotong, Lin, Zhouchen, Zhang, Muhan
Format:	Preprint
Publié:	2026
Sujets:	Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Batching BPE Tokenization Merges
par: Morgan, Alexander P.
Publié: (2024)

Parrot Mind: Towards Explaining the Complex Task Reasoning of Pretrained Large Language Models with Template-Content Structure
par: Yang, Haotong, et autres
Publié: (2023)

BlockBPE: Parallel BPE Tokenization
par: You, Amos
Publié: (2025)

AdaptBPE: From General Purpose to Specialized Tokenizers
par: Liyanage, Vijini, et autres
Publié: (2026)

From Characters to Tokens: Dynamic Grouping with Hierarchical BPE
par: Dolga, Rares, et autres
Publié: (2025)

Number Cookbook: Number Understanding of Language Models and How to Improve It
par: Yang, Haotong, et autres
Publié: (2024)

Constructing a BPE Tokenization DFA
par: Berglund, Martin, et autres
Publié: (2024)

Byte BPE Tokenization as an Inverse string Homomorphism
par: Geng, Saibo, et autres
Publié: (2024)

Proof-RM: A Scalable and Generalizable Reward Model for Math Proof
par: Yang, Haotong, et autres
Publié: (2026)

Scaffold-BPE: Enhancing Byte Pair Encoding for Large Language Models with Simple and Effective Scaffold Token Removal
par: Lian, Haoran, et autres
Publié: (2024)

GL-Fusion: Rethinking the Combination of Graph Neural Network and Large Language model
par: Yang, Haotong, et autres
Publié: (2024)

BPE Gets Picky: Efficient Vocabulary Refinement During Tokenizer Training
par: Chizhov, Pavel, et autres
Publié: (2024)

GPUTOK: GPU Accelerated Byte Level BPE Tokenization
par: Kadamba, Venu Gopal, et autres
Publié: (2026)

Evaluating Subword Tokenization Techniques for Bengali: A Benchmark Study with BengaliBPE
par: Patwary, Firoj Ahmmed, et autres
Publié: (2025)

Adaptive BPE Tokenization for Enhanced Vocabulary Adaptation in Finetuning Pretrained Language Models
par: Balde, Gunjan, et autres
Publié: (2024)

Rethinking Tokenization for Rich Morphology: The Dominance of Unigram over BPE and Morphological Alignment
par: Vemula, Saketh Reddy, et autres
Publié: (2025)

Data Mixture Inference: What do BPE Tokenizers Reveal about their Training Data?
par: Hayase, Jonathan, et autres
Publié: (2024)

Rethinking Personalization in Large Language Models at the Token Level
par: Zhang, Chenheng, et autres
Publié: (2026)

Train It and Forget It: Merge Lists are Unnecessary for BPE Inference in Language Models
par: Sawada, Tomohiro, et autres
Publié: (2025)

SubTokenTest: A Practical Benchmark for Real-World Sub-token Understanding
par: Hou, Shuyang, et autres
Publié: (2026)

Case-Based or Rule-Based: How Do Transformers Do the Math?
par: Hu, Yi, et autres
Publié: (2024)

MorphBPE: A Morpho-Aware Tokenizer Bridging Linguistic Complexity for Efficient LLM Training Across Morphologies
par: Asgari, Ehsaneddin, et autres
Publié: (2025)

Beyond Single-Task: Robust Multi-Task Length Generalization for LLMs
par: Hu, Yi, et autres
Publié: (2025)

Problematic Tokens: Tokenizer Bias in Large Language Models
par: Yang, Jin, et autres
Publié: (2024)

Disentangling Reasoning Tokens and Boilerplate Tokens For Language Model Fine-tuning
par: Ye, Ziang, et autres
Publié: (2024)

Compressing Sequences in the Latent Embedding Space: $K$-Token Merging for Large Language Models
par: Xu, Zihao, et autres
Publié: (2026)

Rethinking Tokenization: Crafting Better Tokenizers for Large Language Models
par: Yang, Jinbiao
Publié: (2024)

Multiplex Thinking: Reasoning via Token-wise Branch-and-Merge
par: Tang, Yao, et autres
Publié: (2026)

LiteVGGT: Boosting Vanilla VGGT via Geometry-aware Cached Token Merging
par: Shu, Zhijian, et autres
Publié: (2025)

Large Language Model as Token Compressor and Decompressor
par: Li, Wenbing, et autres
Publié: (2026)

Behavior-Equivalent Token: Single-Token Replacement for Long Prompts in LLMs
par: Dong, Jiancheng, et autres
Publié: (2025)

Bit-level BPE: Below the byte boundary
par: Moon, Sangwhan, et autres
Publié: (2025)

Wait, We Don't Need to "Wait"! Removing Thinking Tokens Improves Reasoning Efficiency
par: Wang, Chenlong, et autres
Publié: (2025)

InfoMerge: Information-aware Token Compression for Efficient Video Large Language Models
par: Liu, Xinxin, et autres
Publié: (2026)

The Token Tax: Systematic Bias in Multilingual Tokenization
par: Lundin, Jessica M., et autres
Publié: (2025)

Targeted Remasking: Replacing Token Editing with Token-to-Mask Refinement in Discrete Diffusion Language Models
par: Yao, Lin
Publié: (2026)

AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruning
par: Zhong, Yiwu, et autres
Publié: (2024)

Mid-Think: Training-Free Intermediate-Budget Reasoning via Token-Level Triggers
par: Yang, Wang, et autres
Publié: (2026)

D$^{3}$ToM: Decider-Guided Dynamic Token Merging for Accelerating Diffusion MLLMs
par: Chang, Shuochen, et autres
Publié: (2025)

Large Language Models Still Face Challenges in Multi-Hop Reasoning with External Knowledge
par: Zhang, Haotong
Publié: (2024)