Cover Image

Multilingual Tokenization through the Lens of Indian Languages: Challenges and Insights

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Karthika, N J, Brahma, Maharaj, Saluja, Rohit, Ramakrishnan, Ganesh, Desarkar, Maunendra Sankar
Format:	Preprint
Published:	2025
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

MorphTok: Morphologically Grounded Tokenization for Indian Languages
by: Brahma, Maharaj, et al.
Published: (2025)

NLIP_Lab-IITH Multilingual MT System for WAT24 MT Shared Task
by: Brahma, Maharaj, et al.
Published: (2024)

DIWALI: Diversity and Inclusivity aWare cuLture specific Items for India: Dataset and Assessment of LLMs for Cultural Text Adaptation in Indian Context
by: Sahoo, Pramit, et al.
Published: (2025)

NLIP_Lab-IITH Low-Resource MT System for WMT24 Indic MT Shared Task
by: Sahoo, Pramit, et al.
Published: (2024)

The Art of Breaking Words: Rethinking Multilingual Tokenizer Design
by: Thakur, Aamod, et al.
Published: (2025)

BoK: Introducing Bag-of-Keywords Loss for Interpretable Dialogue Response Generation
by: Dey, Suvodip, et al.
Published: (2025)

AyurParam: A State-of-the-Art Bilingual Language Model for Ayurveda
by: Nauman, Mohd, et al.
Published: (2025)

Unsupervised Domain Adaptation with Global and Local Graph Neural Networks in Limited Labeled Data Scenario: Application to Disaster Management
by: Ghosh, Samujjwal, et al.
Published: (2021)

Transformer based Multitask Learning for Image Captioning and Object Detection
by: Basak, Debolena, et al.
Published: (2024)

CharSpan: Utilizing Lexical Similarity to Enable Zero-Shot Machine Translation for Extremely Low-resource Languages
by: Maurya, Kaushal Kumar, et al.
Published: (2023)

LEVOS: Leveraging Vocabulary Overlap with Sanskrit to Generate Technical Lexicons in Indian Languages
by: J, Karthika N, et al.
Published: (2024)

A Three-Pronged Approach to Cross-Lingual Adaptation with Multilingual LLMs
by: Singh, Vaibhav, et al.
Published: (2024)

Semantically Cohesive Word Grouping in Indian Languages
by: Karthika, N J, et al.
Published: (2025)

LexGen: Domain-aware Multilingual Lexicon Generation
by: Maheshwari, Ayush, et al.
Published: (2024)

Samasāmayik: A Parallel Dataset for Hindi-Sanskrit Machine Translation
by: Karthika, N J, et al.
Published: (2026)

INDIC DIALECT: A Multi Task Benchmark to Evaluate and Translate in Indian Language Dialects
by: Sharma, Tarun, et al.
Published: (2026)

Intent Aware Context Retrieval for Multi-Turn Agricultural Question Answering
by: Vijayvargia, Abhay, et al.
Published: (2025)

Indic-TunedLens: Interpreting Multilingual Models in Indian Languages
by: Panchal, Mihir, et al.
Published: (2026)

Dynamic Multi-Expert Projectors with Stabilized Routing for Multilingual Speech Recognition
by: Pandey, Isha, et al.
Published: (2026)

Improving Video Question Answering through query-based frame selection
by: Patil, Himanshu, et al.
Published: (2026)

A2TTS: TTS for Low Resource Indian Languages
by: Bhadoriya, Ayush Singh, et al.
Published: (2025)

Lens: Rethinking Multilingual Enhancement for Large Language Models
by: Zhao, Weixiang, et al.
Published: (2024)

MHQA: A Diverse, Knowledge Intensive Mental Health Question Answering Challenge for Language Models
by: Racha, Suraj, et al.
Published: (2025)

StructFormer: Document Structure-based Masked Attention and its Impact on Language Model Pre-Training
by: Ponkshe, Kaustubh, et al.
Published: (2024)

Tokenization Matters! Degrading Large Language Models through Challenging Their Tokenization
by: Wang, Dixuan, et al.
Published: (2024)

One Tokenizer To Rule Them All: Emergent Language Plasticity via Multilingual Tokenizers
by: Abagyan, Diana, et al.
Published: (2025)

Accelerating Multilingual Language Model for Excessively Tokenized Languages
by: Hong, Jimin, et al.
Published: (2024)

How Language Directions Align with Token Geometry in Multilingual LLMs
by: Kim, JaeSeong, et al.
Published: (2025)

TASE: Token Awareness and Structured Evaluation for Multilingual Language Models
by: Zhao, Chenzhuo, et al.
Published: (2025)

Multilingual Text Style Transfer: Datasets & Models for Indian Languages
by: Mukherjee, Sourabrata, et al.
Published: (2024)

Déjà Vu: Multilingual LLM Evaluation through the Lens of Machine Translation Evaluation
by: Kreutzer, Julia, et al.
Published: (2025)

Beyond Literal Token Overlap: Token Alignability for Multilinguality
by: Hämmerl, Katharina, et al.
Published: (2025)

DICTDIS: Dictionary Constrained Disambiguation for Improved NMT
by: Maheshwari, Ayush, et al.
Published: (2022)

Hybrid Tokenization Strategy for DNA Language Model using Byte Pair Encoding and K-MER Methods
by: Sapkota, Ganesh, et al.
Published: (2025)

Multilingual Jailbreak Challenges in Large Language Models
by: Deng, Yue, et al.
Published: (2023)

Enhancing Whisper's Accuracy and Speed for Indian Languages through Prompt-Tuning and Tokenization
by: Tripathi, Kumud, et al.
Published: (2024)

Building Robust and Scalable Multilingual ASR for Indian Languages
by: Gangwar, Arjun, et al.
Published: (2025)

Evaluating Tokenizer Performance of Large Language Models Across Official Indian Languages
by: Tamang, S., et al.
Published: (2024)

The Token Tax: Systematic Bias in Multilingual Tokenization
by: Lundin, Jessica M., et al.
Published: (2025)

Multilingual European Language Models: Benchmarking Approaches and Challenges
by: Barth, Fabio, et al.
Published: (2025)