Cover Image

Multi-word Tokenization for Sequence Compression

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Gee, Leonidas, Rigutini, Leonardo, Ernandes, Marco, Zugarini, Andrea
Format:	Preprint
Published:	2024
Subjects:	Computation and Language Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Fast Vocabulary Transfer for Language Model Compression
by: Gee, Leonidas, et al.
Published: (2024)

BUSTER: a "BUSiness Transaction Entity Recognition" dataset
by: Zugarini, Andrea, et al.
Published: (2024)

Are Compressed Language Models Less Subgroup Robust?
by: Gee, Leonidas, et al.
Published: (2024)

An energy-based comparative analysis of common approaches to text classification in the Legal domain
by: Gultekin, Sinan, et al.
Published: (2023)

Show Less, Instruct More: Enriching Prompts with Definitions and Guidelines for Zero-Shot NER
by: Zamai, Andrew, et al.
Published: (2024)

SLIMER-IT: Zero-Shot NER on Italian Language
by: Zamai, Andrew, et al.
Published: (2024)

Neural paraphrasing by automatically crawled and aligned sentence pairs
by: Globo, Achille, et al.
Published: (2024)

Lossless Token Sequence Compression via Meta-Tokens
by: Harvill, John, et al.
Published: (2025)

Clue-Instruct: Text-Based Clue Generation for Educational Crossword Puzzles
by: Zugarini, Andrea, et al.
Published: (2024)

Multilingual Training and Evaluation Resources for Vision-Language Models
by: Baiamonte, Daniela, et al.
Published: (2026)

From Token to Token Pair: Efficient Prompt Compression for Large Language Models in Clinical Prediction
by: Zhu, Mingcheng, et al.
Published: (2026)

A novel integrated industrial approach with cobots in the age of industry 4.0 through conversational interaction and computer vision
by: Pazienza, Andrea, et al.
Published: (2024)

Temporal Tokenization Strategies for Event Sequence Modeling with Large Language Models
by: Liu, Zefang, et al.
Published: (2025)

PairAlign: A Framework for Sequence Tokenization via Self-Alignment with Applications to Audio Tokenization
by: Banerjee, Adhiraj, et al.
Published: (2026)

MultiTok: Variable-Length Tokenization for Efficient LLMs Adapted from LZW Compression
by: Elias, Noel, et al.
Published: (2024)

zip2zip: Inference-Time Adaptive Tokenization via Online Compression
by: Geng, Saibo, et al.
Published: (2025)

Multitask Kernel-based Learning with First-Order Logic Constraints
by: Diligenti, Michelangelo, et al.
Published: (2023)

ByteFlow: Language Modeling through Adaptive Byte Compression without a Tokenizer
by: Deng, Chunyuan, et al.
Published: (2026)

Lossless Compression of Large Language Model-Generated Text via Next-Token Prediction
by: Mao, Yu, et al.
Published: (2025)

Attamba: Attending To Multi-Token States
by: Akhauri, Yash, et al.
Published: (2024)

Multi-Token Prediction via Self-Distillation
by: Kirchenbauer, John, et al.
Published: (2026)

Attention with Trained Embeddings Provably Selects Important Tokens
by: Wu, Diyuan, et al.
Published: (2025)

Deep learning models for representing out-of-vocabulary words
by: Lochter, Johannes V., et al.
Published: (2020)

Nectar: Neural Estimation of Cached-Token Attention via Regression
by: Monteiro, João, et al.
Published: (2026)

Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performance
by: Goldman, Omer, et al.
Published: (2024)

Better Prompt Compression Without Multi-Layer Perceptrons
by: Honig, Edouardo, et al.
Published: (2025)

Optimized Multi-Token Joint Decoding with Auxiliary Model for LLM Inference
by: Qin, Zongyue, et al.
Published: (2024)

Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potential
by: Samragh, Mohammad, et al.
Published: (2025)

Critical biblical studies via word frequency analysis: unveiling text authorship
by: Faigenbaum-Golovin, Shira, et al.
Published: (2024)

One Size Does Not Fit All: Token-Wise Adaptive Compression for KV Cache
by: Lu, Liming, et al.
Published: (2026)

TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior
by: Altıntaş, Gül Sena, et al.
Published: (2025)

Self-Distillation for Multi-Token Prediction
by: Zhao, Guoliang, et al.
Published: (2026)

TokenShapley: Token Level Context Attribution with Shapley Value
by: Xiao, Yingtai, et al.
Published: (2025)

Token Distillation: Attention-aware Input Embeddings For New Tokens
by: Dobler, Konstantin, et al.
Published: (2025)

X-Token: Projection-Guided Cross-Tokenizer Knowledge Distillation
by: Sreenivas, Sharath Turuvekere, et al.
Published: (2026)

Automatic Differential Diagnosis using Transformer-Based Multi-Label Sequence Classification
by: Sadi, Abu Adnan, et al.
Published: (2024)

Data Augmentation and Transfer Learning Approaches Applied to Facial Expressions Recognition
by: Randellini, Enrico, et al.
Published: (2024)

TensorLLM: Tensorising Multi-Head Attention for Enhanced Reasoning and Compression in LLMs
by: Gu, Yuxuan, et al.
Published: (2025)

Transforming Chatbot Text: A Sequence-to-Sequence Approach
by: Reddy, Natesh, et al.
Published: (2025)

Token Sparse Attention: Efficient Long-Context Inference with Interleaved Token Selection
by: Jo, Dongwon, et al.
Published: (2026)