Image de couverture de livre

The Token Tax: Systematic Bias in Multilingual Tokenization

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Lundin, Jessica M., Zhang, Ada, Karim, Nihal, Louzan, Hamza, Wei, Victor, Adelani, David, Carroll, Cody
Format:	Preprint
Publié:	2025
Sujets:	Computation and Language Artificial Intelligence
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

No Text Needed: Forecasting MT Quality and Inequity from Fertility and Metadata
par: Lundin, Jessica M., et autres
Publié: (2025)

The Script Tax: Measuring Tokenization-Driven Efficiency and Latency Disparities in Multilingual Language Models
par: Dixit, Aradhya, et autres
Publié: (2026)

The Art of Breaking Words: Rethinking Multilingual Tokenizer Design
par: Thakur, Aamod, et autres
Publié: (2025)

Accelerating Multilingual Language Model for Excessively Tokenized Languages
par: Hong, Jimin, et autres
Publié: (2024)

Fleurs-SLU: A Massively Multilingual Benchmark for Spoken Language Understanding
par: Schmidt, Fabian David, et autres
Publié: (2025)

Tokenization and Representation Biases in Multilingual Models on Dialectal NLP Tasks
par: Kanjirangat, Vani, et autres
Publié: (2025)

Natural language processing for African languages
par: Adelani, David Ifeoluwa
Publié: (2025)

Token Homogenization under Positional Bias
par: Yusupov, Viacheslav, et autres
Publié: (2025)

Revisiting Graph-Tokenizing Large Language Models: A Systematic Evaluation of Graph Token Understanding
par: Zhang, Zhongjian, et autres
Publié: (2026)

Understanding and Mitigating Tokenization Bias in Language Models
par: Phan, Buu, et autres
Publié: (2024)

Technical Report: Impact of Position Bias on Language Models in Token Classification
par: Amor, Mehdi Ben, et autres
Publié: (2023)

Thinking Tokens for Language Modeling
par: Herel, David, et autres
Publié: (2024)

Beyond Fertility: Analyzing STRR as a Metric for Multilingual Tokenization Evaluation
par: Nayeem, Mir Tafseer, et autres
Publié: (2025)

A Peek into Token Bias: Large Language Models Are Not Yet Genuine Reasoners
par: Jiang, Bowen, et autres
Publié: (2024)

On the Bias of Next-Token Predictors Toward Systematically Inefficient Reasoning: A Shortest-Path Case Study
par: Alberghi, Riccardo, et autres
Publié: (2025)

DIVERS-Bench: Evaluating Language Identification Across Domain Shifts and Code-Switching
par: Ojo, Jessica, et autres
Publié: (2025)

State over Tokens: Characterizing the Role of Reasoning Tokens
par: Levy, Mosh, et autres
Publié: (2025)

One Token Is Enough: Improving Diffusion Language Models with a Sink Token
par: Zhang, Zihou, et autres
Publié: (2026)

From Guidelines to Guarantees: A Graph-Based Evaluation Harness for Domain-Specific Evaluation of LLMs
par: Lundin, Jessica M., et autres
Publié: (2025)

TokenSeek: Memory Efficient Fine Tuning via Instance-Aware Token Ditching
par: Zeng, Runjia, et autres
Publié: (2026)

Tokenization Matters! Degrading Large Language Models through Challenging Their Tokenization
par: Wang, Dixuan, et autres
Publié: (2024)

Translation as a Scalable Proxy for Multilingual Evaluation
par: Issaka, Sheriff, et autres
Publié: (2026)

ANGOFA: Leveraging OFA Embedding Initialization and Synthetic Data for Angolan Language Model
par: Quinjica, Osvaldo Luamba, et autres
Publié: (2024)

TokenButler: Token Importance is Predictable
par: Akhauri, Yash, et autres
Publié: (2025)

SemToken: Semantic-Aware Tokenization for Efficient Long-Context Language Modeling
par: Liu, Dong, et autres
Publié: (2025)

TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching
par: Nguyen, Truong, et autres
Publié: (2026)

Rubrics to Tokens: Bridging Response-level Rubrics and Token-level Rewards in Instruction Following Tasks
par: Xu, Tianze, et autres
Publié: (2026)

LLMs are Not Just Next Token Predictors
par: Downes, Stephen M., et autres
Publié: (2024)

Token-level Direct Preference Optimization
par: Zeng, Yongcheng, et autres
Publié: (2024)

CompactRAG: Reducing LLM Calls and Token Overhead in Multi-Hop Question Answering
par: Yang, Hao, et autres
Publié: (2026)

An Expert is Worth One Token: Synergizing Multiple Expert LLMs as Generalist via Expert Token Routing
par: Chai, Ziwei, et autres
Publié: (2024)

Token-Guard: Towards Token-Level Hallucination Control via Self-Checking Decoding
par: Zhu, Yifan, et autres
Publié: (2026)

Tokenization Constraints in LLMs: A Study of Symbolic and Arithmetic Reasoning Limits
par: Zhang, Xiang, et autres
Publié: (2025)

Reinforcement Learning with Token-level Feedback for Controllable Text Generation
par: Li, Wendi, et autres
Publié: (2024)

Direct Multi-Token Decoding
par: Luo, Xuan, et autres
Publié: (2025)

Batching BPE Tokenization Merges
par: Morgan, Alexander P.
Publié: (2024)

Date Fragments: A Hidden Bottleneck of Tokenization for Temporal Reasoning
par: Bhatia, Gagan, et autres
Publié: (2025)

Thunder-Tok: Minimizing Tokens per Word in Tokenizing Korean Texts for Generative Language Models
par: Cho, Gyeongje, et autres
Publié: (2025)

Token Signature: Predicting Chain-of-Thought Gains with Token Decoding Feature in Large Language Models
par: Liu, Peijie, et autres
Publié: (2025)

Informed Routing in LLMs: Smarter Token-Level Computation for Faster Inference
par: Han, Chao, et autres
Publié: (2025)