Cover Image

The Script Tax: Measuring Tokenization-Driven Efficiency and Latency Disparities in Multilingual Language Models

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Dixit, Aradhya, Dixit, Shreem
Format:	Preprint
Published:	2026
Subjects:	Computation and Language Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

PCN-Rec: Agentic Proof-Carrying Negotiation for Reliable Governance-Constrained Recommendation
by: Dixit, Aradhya, et al.
Published: (2026)

Semantic Event Graphs for Long-Form Video Question Answering
by: Dixit, Aradhya, et al.
Published: (2026)

The Token Tax: Systematic Bias in Multilingual Tokenization
by: Lundin, Jessica M., et al.
Published: (2025)

Project Aletheia: Verifier-Guided Distillation of Backtracking for Small Language Models
by: Dixit, Aradhya, et al.
Published: (2026)

Accelerating Multilingual Language Model for Excessively Tokenized Languages
by: Hong, Jimin, et al.
Published: (2024)

Evaluating Self-Correcting Vision Agents Through Quantitative and Qualitative Metrics
by: Dixit, Aradhya
Published: (2026)

What Works for 'Lost-in-the-Middle' in LLMs? A Study on GM-Extract and Mitigations
by: Gupte, Mihir, et al.
Published: (2025)

Does Visual Rendering Bypass Tokenization? Investigating Script-Tokenizer Misalignment in Pixel-Based Language Models
by: Susanto, Lucky, et al.
Published: (2026)

Script Sensitivity: Benchmarking Language Models on Unicode, Romanized and Mixed-Script Sinhala
by: Rajapakse, Minuri, et al.
Published: (2026)

Faux Polyglot: A Study on Information Disparity in Multilingual Large Language Models
by: Sharma, Nikhil, et al.
Published: (2024)

Prompting with Phonemes: Enhancing LLMs' Multilinguality for Non-Latin Script Languages
by: Nguyen, Hoang H, et al.
Published: (2024)

Emergent Languages in Populations of Language Model Agents: From Token Efficiency to Oversight Evasion
by: Beltoft, Stine Lyngsø, et al.
Published: (2026)

Tokenization and Representation Biases in Multilingual Models on Dialectal NLP Tasks
by: Kanjirangat, Vani, et al.
Published: (2025)

Evidence-Guided Schema Normalization for Temporal Tabular Reasoning
by: Thanga, Ashish, et al.
Published: (2025)

VocalNet-M2: Advancing Low-Latency Spoken Language Modeling via Integrated Multi-Codebook Tokenization and Multi-Token Prediction
by: Wang, Yuhao, et al.
Published: (2025)

Language Imbalance Driven Rewarding for Multilingual Self-improving
by: Yang, Wen, et al.
Published: (2024)

MultiScript30k: Leveraging Multilingual Embeddings to Extend Cross Script Parallel Data
by: Driggers-Ellis, Christopher, et al.
Published: (2025)

No Universal Prompt: Unifying Reasoning through Adaptive Prompting for Temporal Table Reasoning
by: Rajgaria, Abhishek, et al.
Published: (2025)

Latency and Token-Aware Test-Time Compute
by: Huang, Jenny Y., et al.
Published: (2025)

Enhancing Temporal Understanding in LLMs for Semi-structured Tables
by: Deng, Irwin, et al.
Published: (2024)

Language Models and Logic Programs for Trustworthy Tax Reasoning
by: Jurayj, William, et al.
Published: (2025)

The Art of Breaking Words: Rethinking Multilingual Tokenizer Design
by: Thakur, Aamod, et al.
Published: (2025)

Multilingual Pretraining for Pixel Language Models
by: Kesen, Ilker, et al.
Published: (2025)

NusaAksara: A Multimodal and Multilingual Benchmark for Preserving Indonesian Indigenous Scripts
by: Adilazuarda, Muhammad Farid, et al.
Published: (2025)

Thinking Tokens for Language Modeling
by: Herel, David, et al.
Published: (2024)

Multilingual Collaborative Defense for Large Language Models
by: Li, Hongliang, et al.
Published: (2025)

Cross-lingual Editing in Multilingual Language Models
by: Beniwal, Himanshu, et al.
Published: (2024)

The Roles of English in Evaluating Multilingual Language Models
by: Poelman, Wessel, et al.
Published: (2024)

SUTRA: Scalable Multilingual Language Model Architecture
by: Bendale, Abhijit, et al.
Published: (2024)

Breaking Language Barriers: Equitable Performance in Multilingual Language Models
by: Nagar, Tanay, et al.
Published: (2025)

All Languages Matter: On the Multilingual Safety of Large Language Models
by: Wang, Wenxuan, et al.
Published: (2023)

Tokenization Matters! Degrading Large Language Models through Challenging Their Tokenization
by: Wang, Dixuan, et al.
Published: (2024)

MYTE: Morphology-Driven Byte Encoding for Better and Fairer Multilingual Language Modeling
by: Limisiewicz, Tomasz, et al.
Published: (2024)

BiasLab: A Multilingual, Dual-Framing Framework for Robust Measurement of Output-Level Bias in Large Language Models
by: Guey, William, et al.
Published: (2026)

Is It Good Data for Multilingual Instruction Tuning or Just Bad Multilingual Evaluation for Large Language Models?
by: Chen, Pinzhen, et al.
Published: (2024)

Nile-Chat: Egyptian Language Models for Arabic and Latin Scripts
by: Shang, Guokan, et al.
Published: (2025)

Measuring Moral LLM Responses in Multilingual Capacities
by: Basu, Kimaya, et al.
Published: (2025)

One Token Is Enough: Improving Diffusion Language Models with a Sink Token
by: Zhang, Zihou, et al.
Published: (2026)

Taxation Perspectives from Large Language Models: A Case Study on Additional Tax Penalties
by: Choi, Eunkyung, et al.
Published: (2025)

Cross-Lingual Activation Steering for Multilingual Language Models
by: Pokharel, Rhitabrat, et al.
Published: (2026)