Cover Image

Accelerating Multilingual Language Model for Excessively Tokenized Languages

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Hong, Jimin, Lee, Gibbeum, Cho, Jaewoong
Format:	Preprint
Published:	2024
Subjects:	Computation and Language Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Latent Paraphrasing: Perturbation on Layers Improves Knowledge Injection in Language Models
by: Kang, Minki, et al.
Published: (2024)

T1: Tool-integrated Verification for Test-time Compute Scaling in Small Language Models
by: Kang, Minki, et al.
Published: (2025)

Thunder-Tok: Minimizing Tokens per Word in Tokenizing Korean Texts for Generative Language Models
by: Cho, Gyeongje, et al.
Published: (2025)

SGuard-v1: Safety Guardrail for Large Language Models
by: Lee, JoonHo, et al.
Published: (2025)

Language over Content: Tracing Cultural Understanding in Multilingual Large Language Models
by: Cho, Seungho, et al.
Published: (2025)

Pruning and Distilling Mixture-of-Experts into Dense Language Models
by: Kim, Junhyuck, et al.
Published: (2026)

Bridging the Gap between Expert and Language Models: Concept-guided Chess Commentary Generation and Evaluation
by: Kim, Jaechang, et al.
Published: (2024)

The Script Tax: Measuring Tokenization-Driven Efficiency and Latency Disparities in Multilingual Language Models
by: Dixit, Aradhya, et al.
Published: (2026)

Distilling LLM Agent into Small Models with Retrieval and Code Tools
by: Kang, Minki, et al.
Published: (2025)

Predictive Pipelined Decoding: A Compute-Latency Trade-off for Exact LLM Decoding
by: Yang, Seongjun, et al.
Published: (2023)

SUTRA: Scalable Multilingual Language Model Architecture
by: Bendale, Abhijit, et al.
Published: (2024)

Optimizing Language Augmentation for Multilingual Large Language Models: A Case Study on Korean
by: Choi, ChangSu, et al.
Published: (2024)

Multilingual Pretraining for Pixel Language Models
by: Kesen, Ilker, et al.
Published: (2025)

Thinking Tokens for Language Modeling
by: Herel, David, et al.
Published: (2024)

Breaking Language Barriers: Equitable Performance in Multilingual Language Models
by: Nagar, Tanay, et al.
Published: (2025)

All Languages Matter: On the Multilingual Safety of Large Language Models
by: Wang, Wenxuan, et al.
Published: (2023)

The Token Tax: Systematic Bias in Multilingual Tokenization
by: Lundin, Jessica M., et al.
Published: (2025)

TLPO: Token-Level Policy Optimization for Mitigating Language Confusion in Large Language Models
by: Choo, Jinho, et al.
Published: (2026)

Direct Token Optimization: A Self-contained Approach to Large Language Model Unlearning
by: Lee, Hong kyu, et al.
Published: (2025)

On Epistemic Uncertainty of Visual Tokens for Object Hallucinations in Large Vision-Language Models
by: Seo, Hoigi, et al.
Published: (2025)

Cross-lingual Editing in Multilingual Language Models
by: Beniwal, Himanshu, et al.
Published: (2024)

The Roles of English in Evaluating Multilingual Language Models
by: Poelman, Wessel, et al.
Published: (2024)

Multilingual Collaborative Defense for Large Language Models
by: Li, Hongliang, et al.
Published: (2025)

Design of reliable technology valuation model with calibrated machine learning of patent indicators
by: Lee, Seunghyun, et al.
Published: (2024)

Early screening of potential breakthrough technologies with enhanced interpretability: A patent-specific hierarchical attention network model
by: Choi, Jaewoong, et al.
Published: (2024)

Chimera: A Lossless Decoding Method for Accelerating Large Language Models Inference by Fusing all Tokens
by: Zeng, Ziqian, et al.
Published: (2024)

Focusing on Language: Revealing and Exploiting Language Attention Heads in Multilingual Large Language Models
by: Liu, Xin, et al.
Published: (2025)

From Language Models over Tokens to Language Models over Characters
by: Vieira, Tim, et al.
Published: (2024)

Accelerating Large Language Model Pretraining via LFR Pedagogy: Learn, Focus, and Review
by: Prakriya, Neha, et al.
Published: (2024)

Tokenization Matters! Degrading Large Language Models through Challenging Their Tokenization
by: Wang, Dixuan, et al.
Published: (2024)

On the Fallacy of Global Token Perplexity in Spoken Language Model Evaluation
by: Hsu, Chan-Jan, et al.
Published: (2026)

LinguaSafe: A Comprehensive Multilingual Safety Benchmark for Large Language Models
by: Ning, Zhiyuan, et al.
Published: (2025)

How do Large Language Models Handle Multilingualism?
by: Zhao, Yiran, et al.
Published: (2024)

Multilingual Performance Biases of Large Language Models in Education
by: Gupta, Vansh, et al.
Published: (2025)

Cross-Lingual Activation Steering for Multilingual Language Models
by: Pokharel, Rhitabrat, et al.
Published: (2026)

Multilingual Training and Evaluation Resources for Vision-Language Models
by: Baiamonte, Daniela, et al.
Published: (2026)

A Shared Geometry of Difficulty in Multilingual Language Models
by: Civelli, Stefano, et al.
Published: (2026)

Tokenization and Representation Biases in Multilingual Models on Dialectal NLP Tasks
by: Kanjirangat, Vani, et al.
Published: (2025)

Compositional Steering of Large Language Models with Steering Tokens
by: Radevski, Gorjan, et al.
Published: (2026)

SemToken: Semantic-Aware Tokenization for Efficient Long-Context Language Modeling
by: Liu, Dong, et al.
Published: (2025)