LoPT: Lossless Parallel Tokenization Acceleration for Long Context Inference of Large Language Model
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911253380726784 |
|---|---|
| author | Shao, Wei Zheng, Lingchao Wang, Pengyu Zheng, Peizhen Li, Jun Fan, Yuwei |
| author_facet | Shao, Wei Zheng, Lingchao Wang, Pengyu Zheng, Peizhen Li, Jun Fan, Yuwei |
| contents | Long context inference scenarios have become increasingly important for large language models, yet they introduce significant computational latency. While prior research has optimized long-sequence inference through operators, model architectures, and system frameworks, tokenization remains an overlooked bottleneck. Existing parallel tokenization methods accelerate processing through text segmentation and multi-process tokenization, but they suffer from inconsistent results due to boundary artifacts that occur after merging. To address this, we propose LoPT, a novel Lossless Parallel Tokenization framework that ensures output identical to standard sequential tokenization. Our approach employs character-position-based matching and dynamic chunk length adjustment to align and merge tokenized segments accurately. Extensive experiments across diverse long-text datasets demonstrate that LoPT achieves significant speedup while guaranteeing lossless tokenization. We also provide theoretical proof of consistency and comprehensive analytical studies to validate the robustness of our method. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_04952 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LoPT: Lossless Parallel Tokenization Acceleration for Long Context Inference of Large Language Model Shao, Wei Zheng, Lingchao Wang, Pengyu Zheng, Peizhen Li, Jun Fan, Yuwei Computation and Language Long context inference scenarios have become increasingly important for large language models, yet they introduce significant computational latency. While prior research has optimized long-sequence inference through operators, model architectures, and system frameworks, tokenization remains an overlooked bottleneck. Existing parallel tokenization methods accelerate processing through text segmentation and multi-process tokenization, but they suffer from inconsistent results due to boundary artifacts that occur after merging. To address this, we propose LoPT, a novel Lossless Parallel Tokenization framework that ensures output identical to standard sequential tokenization. Our approach employs character-position-based matching and dynamic chunk length adjustment to align and merge tokenized segments accurately. Extensive experiments across diverse long-text datasets demonstrate that LoPT achieves significant speedup while guaranteeing lossless tokenization. We also provide theoretical proof of consistency and comprehensive analytical studies to validate the robustness of our method. |
| title | LoPT: Lossless Parallel Tokenization Acceleration for Long Context Inference of Large Language Model |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2511.04952 |