Cover Image

MultiTok: Variable-Length Tokenization for Efficient LLMs Adapted from LZW Compression

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Elias, Noel, Esfahanizadeh, Homa, Kale, Kaan, Vishwanath, Sriram, Medard, Muriel
Format:	Preprint
Published:	2024
Subjects:	Computation and Language Information Theory Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

TexShape: Information Theoretic Sentence Embedding for Language Models
by: Kale, Kaan, et al.
Published: (2024)

Successive Refinement in Large-Scale Computation: Advancing Model Inference Applications
by: Esfahanizadeh, Homa, et al.
Published: (2024)

On the Benefits of Coding for Network Slicing
by: Esfahanizadeh, Homa, et al.
Published: (2024)

Multi-level Reliability Interface for Semantic Communications over Wireless Networks
by: Tung, Tze-Yang, et al.
Published: (2024)

Precoding-Oriented CSI Feedback Design with Mutual Information Regularized VQ-VAE
by: Chen, Xi, et al.
Published: (2026)

Learning Obfuscations Of LLM Embedding Sequences: Stained Glass Transform
by: Roberts, Jay, et al.
Published: (2025)

Efficient Learned Data Compression via Dual-Stream Feature Decoupling
by: Ma, Huidong, et al.
Published: (2026)

From Tokens to Thoughts: How LLMs and Humans Trade Compression for Meaning
by: Shani, Chen, et al.
Published: (2025)

CRC Codes as Error Correction Codes
by: An, Wei, et al.
Published: (2021)

Optimum Peer-Turbo: A Scalable and Efficient Solution for P2P Broadcasting
by: Médard, Muriel, et al.
Published: (2026)

Group Probability Decoding of Turbo Product Codes over Higher-Order Fields
by: Rapp, Lukas, et al.
Published: (2025)

Soft-Output Successive Cancellation List Decoding
by: Yuan, Peihong, et al.
Published: (2024)

SOGRAND Assisted Guesswork Reduction
by: Rapp, Lukas, et al.
Published: (2025)

The Linear Reliability Channel
by: Mariona, Alexander, et al.
Published: (2025)

Near-Optimal Generalized Decoding of Polar-like Codes
by: Yuan, Peihong, et al.
Published: (2024)

Turbo product decoding of cubic tensor codes
by: Khalifeh, Sarah, et al.
Published: (2024)

Leveraging Code Structure to Improve Soft Output for GRAND, GCD, OSD, and SCL
by: Feng, Jiewei, et al.
Published: (2025)

Effective Context in Transformers: An Analysis of Fragmentation and Tokenization
by: Fesharaki, Amirmehdi Jafari, et al.
Published: (2026)

Efficient Soft-Output Guessing for Enhanced Quantum Tanner Code Decoding
by: Rapp, Lukas, et al.
Published: (2026)

A Monotone Circuit Construction for Individually-Secure Multi-Secret Sharing
by: Bass, Cailyn, et al.
Published: (2024)

Pricing Innovation Under Latency Constraints: A Mean-Field Analysis of Coded Payload Delivery
by: Médard, Muriel, et al.
Published: (2026)

An Enhanced Text Compression Approach Using Transformer-based Language Models
by: Rahman, Chowdhury Mofizur, et al.
Published: (2024)

Joint Error Correction and Fading Channel Estimation Enhancement Leveraging GRAND
by: Wiame, Charles, et al.
Published: (2025)

Error correction in interference-limited wireless systems
by: Wiame, Charles, et al.
Published: (2024)

Guessing random additive noise decoding with symbol reliability information (SRGRAND)
by: Duffy, Ken R., et al.
Published: (2019)

Variable-Length Semantic IDs for Recommender Systems
by: Khrylchenko, Kirill
Published: (2026)

Fundamental Limits of Prompt Compression: A Rate-Distortion Framework for Black-Box Language Models
by: Nagle, Alliot, et al.
Published: (2024)

Modular Representation Compression: Adapting LLMs for Efficient and Effective Recommendations
by: Xi, Yunjia, et al.
Published: (2026)

Language Modeling Is Compression
by: Delétang, Grégoire, et al.
Published: (2023)

Soft-output (SO) GRAND and Iterative Decoding to Outperform LDPCs
by: Yuan, Peihong, et al.
Published: (2023)

A Balanced Tree Transformation to Reduce GRAND Queries
by: Rapp, Lukas, et al.
Published: (2025)

Using a Single-Parity-Check to Reduce the Guesswork of Guessing Codeword Decoding
by: Griffin, Joseph, et al.
Published: (2024)

Soft-output Guessing Codeword Decoding
by: Duffy, Ken R., et al.
Published: (2024)

Quad Length Codes for Lossless Compression of e4m3
by: Agrawal, Aditya, et al.
Published: (2026)

LINC: An In-Network Coding Approach to Tame Packet Loss in Hybrid Wireless-Fiber Backbones
by: Pit-Claudel, Benoit, et al.
Published: (2025)

Compression Represents Intelligence Linearly
by: Huang, Yuzhen, et al.
Published: (2024)

What Makes the Preferred Thinking Direction for LLMs in Multiple-choice Questions?
by: Zhang, Yizhe, et al.
Published: (2025)

Memorization-Compression Cycles Improve Generalization
by: Yu, Fangyuan
Published: (2025)

Filtering Beats Fine Tuning: A Bayesian Kalman View of In Context Learning in LLMs
by: Kiruluta, Andrew
Published: (2026)

The Benefit of Decoder-Provided Pilots in Highly Dynamic Channels
by: Bodet, Duschia, et al.
Published: (2026)