Cover Image

Large Language Model as Token Compressor and Decompressor

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Li, Wenbing, Wang, Yiran, Song, Zikai, Zhang, Jielei, Zhao, Tianhao, Lin, Junkai, Yang, Wei
Format:	Preprint
Published:	2026
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Problematic Tokens: Tokenizer Bias in Large Language Models
by: Yang, Jin, et al.
Published: (2024)

500xCompressor: Generalized Prompt Compression for Large Language Models
by: Li, Zongqian, et al.
Published: (2024)

The Compressor-Retriever Architecture for Language Model OS
by: Yang, Yuan, et al.
Published: (2024)

Order Matters in Hallucination: Reasoning Order as Benchmark and Reflexive Prompting for Large-Language-Models
by: Xie, Zikai
Published: (2024)

Rethinking Personalization in Large Language Models at the Token Level
by: Zhang, Chenheng, et al.
Published: (2026)

The Rise of Parameter Specialization for Knowledge Storage in Large Language Models
by: Hong, Yihuai, et al.
Published: (2025)

Diving into Mitigating Hallucinations from a Vision Perspective for Large Vision-Language Models
by: Wang, Weihang, et al.
Published: (2025)

Disentangling Language and Culture for Evaluating Multilingual Large Language Models
by: Ying, Jiahao, et al.
Published: (2025)

Keyframe-oriented Vision Token Pruning: Enhancing Efficiency of Large Vision Language Models on Long-Form Video Processing
by: Liu, Yudong, et al.
Published: (2025)

Rethinking Tokenization: Crafting Better Tokenizers for Large Language Models
by: Yang, Jinbiao
Published: (2024)

Adaptable and Reliable Text Classification using Large Language Models
by: Wang, Zhiqiang, et al.
Published: (2024)

Glitch Tokens in Large Language Models: Categorization Taxonomy and Effective Detection
by: Li, Yuxi, et al.
Published: (2024)

Majority Bit-Aware Watermarking For Large Language Models
by: Xu, Jiahao, et al.
Published: (2025)

Cmprsr: Abstractive Token-Level Question-Agnostic Prompt Compressor
by: Zakazov, Ivan, et al.
Published: (2025)

ReTok: Replacing Tokenizer to Enhance Representation Efficiency in Large Language Model
by: Gu, Shuhao, et al.
Published: (2024)

Memory Retrieval and Consolidation in Large Language Models through Function Tokens
by: Zhang, Shaohua, et al.
Published: (2025)

ResT: Reshaping Token-Level Policy Gradients for Tool-Use Large Language Models
by: Lin, Zihan, et al.
Published: (2025)

LFTR: Learning-Free Token Reduction for Multimodal Large Language Models
by: Zhao, Zihui, et al.
Published: (2025)

Large Language Model Bias Mitigation from the Perspective of Knowledge Editing
by: Chen, Ruizhe, et al.
Published: (2024)

SemanticZip: A Pilot Framework for Lossy Text Compression with LLMs as Semantic Decompressors
by: Trukhina, Natalia, et al.
Published: (2026)

Disentangling Reasoning Tokens and Boilerplate Tokens For Language Model Fine-tuning
by: Ye, Ziang, et al.
Published: (2024)

Relative Kinetic Utility for Reasoning-Aware Structural Pruning in Large Language Models
by: Qian, Tianhao
Published: (2026)

CLEAN-EVAL: Clean Evaluation on Contaminated Large Language Models
by: Zhu, Wenhong, et al.
Published: (2023)

Tokenization Matters! Degrading Large Language Models through Challenging Their Tokenization
by: Wang, Dixuan, et al.
Published: (2024)

Beyond Token-Level Policy Gradients for Complex Reasoning with Large Language Models
by: Xu, Mufan, et al.
Published: (2026)

DEEM: Diffusion Models Serve as the Eyes of Large Language Models for Image Perception
by: Luo, Run, et al.
Published: (2024)

Word Recovery in Large Language Models Enables Character-Level Tokenization Robustness
by: Yang, Zhipeng, et al.
Published: (2026)

Remask, Don't Replace: Token-to-Mask Refinement in Diffusion Large Language Models
by: Yao, Lin
Published: (2026)

Can Large Language Models Understand Internet Buzzwords Through User-Generated Content
by: Huang, Chen, et al.
Published: (2025)

How do Large Language Models Handle Multilingualism?
by: Zhao, Yiran, et al.
Published: (2024)

RoleEval: A Bilingual Role Evaluation Benchmark for Large Language Models
by: Shen, Tianhao, et al.
Published: (2023)

ParallelComp: Parallel Long-Context Compressor for Length Extrapolation
by: Xiong, Jing, et al.
Published: (2025)

Knowledge Distillation for Temporal Knowledge Graph Reasoning with Large Language Models
by: Xing, Wang, et al.
Published: (2026)

FTP: A Fine-grained Token-wise Pruner for Large Language Models via Token Routing
by: Li, Zekai, et al.
Published: (2024)

Perception Compressor: A Training-Free Prompt Compression Framework in Long Context Scenarios
by: Tang, Jiwei, et al.
Published: (2024)

AdaPonderLM: Gated Pondering Language Models with Token-Wise Adaptive Depth
by: Song, Shixiang, et al.
Published: (2026)

SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models
by: Zhang, Xin, et al.
Published: (2023)

LoPT: Lossless Parallel Tokenization Acceleration for Long Context Inference of Large Language Model
by: Shao, Wei, et al.
Published: (2025)

Investigating Numerical Translation with Large Language Models
by: Tang, Wei, et al.
Published: (2025)

The Emergence of Abstract Thought in Large Language Models Beyond Any Language
by: Chen, Yuxin, et al.
Published: (2025)