Cover Image

The Art of Breaking Words: Rethinking Multilingual Tokenizer Design

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Thakur, Aamod, Nagpal, Ajay, Savarkar, Atharva, Pundalik, Kundeshwar, Dosi, Siddhesh, Sawarkar, Piyush, Thakur, Viraj, Saluja, Rohit, Desarkar, Maunendra Sankar, Ramakrishnan, Ganesh
Format:	Preprint
Published:	2025
Subjects:	Computation and Language Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

AyurParam: A State-of-the-Art Bilingual Language Model for Ayurveda
by: Nauman, Mohd, et al.
Published: (2025)

Intent Aware Context Retrieval for Multi-Turn Agricultural Question Answering
by: Vijayvargia, Abhay, et al.
Published: (2025)

Multilingual Tokenization through the Lens of Indian Languages: Challenges and Insights
by: Karthika, N J, et al.
Published: (2025)

PARAM-1 BharatGen 2.9B Model
by: Pundalik, Kundeshwar, et al.
Published: (2025)

BhashaBench V1: A Comprehensive Benchmark for the Quadrant of Indic Domains
by: Devane, Vijay, et al.
Published: (2025)

MorphTok: Morphologically Grounded Tokenization for Indian Languages
by: Brahma, Maharaj, et al.
Published: (2025)

NLIP_Lab-IITH Multilingual MT System for WAT24 MT Shared Task
by: Brahma, Maharaj, et al.
Published: (2024)

BoK: Introducing Bag-of-Keywords Loss for Interpretable Dialogue Response Generation
by: Dey, Suvodip, et al.
Published: (2025)

DIWALI: Diversity and Inclusivity aWare cuLture specific Items for India: Dataset and Assessment of LLMs for Cultural Text Adaptation in Indian Context
by: Sahoo, Pramit, et al.
Published: (2025)

NLIP_Lab-IITH Low-Resource MT System for WMT24 Indic MT Shared Task
by: Sahoo, Pramit, et al.
Published: (2024)

Unsupervised Domain Adaptation with Global and Local Graph Neural Networks in Limited Labeled Data Scenario: Application to Disaster Management
by: Ghosh, Samujjwal, et al.
Published: (2021)

Transformer based Multitask Learning for Image Captioning and Object Detection
by: Basak, Debolena, et al.
Published: (2024)

CharSpan: Utilizing Lexical Similarity to Enable Zero-Shot Machine Translation for Extremely Low-resource Languages
by: Maurya, Kaushal Kumar, et al.
Published: (2023)

Explainable AI For Early Detection Of Sepsis
by: Thakur, Atharva, et al.
Published: (2025)

Improving Video Question Answering through query-based frame selection
by: Patil, Himanshu, et al.
Published: (2026)

Mpox Narrative on Instagram: A Labeled Multilingual Dataset of Instagram Posts on Mpox for Sentiment, Hate Speech, and Anxiety Analysis
by: Thakur, Nirmalya
Published: (2024)

Five Years of COVID-19 Discourse on Instagram: A Labeled Instagram Dataset of Over Half a Million Posts for Multilingual Sentiment Analysis
by: Thakur, Nirmalya
Published: (2024)

Nonlocal thermo‐elastic discs: A study of transitional stresses and temperature‐dependent modulus of rigidity
by: Ajay, et al.
Published: (2025)

Enhanced Strength and Thermal Stability in Nacre‐Inspired Polyvinyl Alcohol/Graphene Oxide Composites via Biomass Gasification–Derived Carbon Nanomaterials
by: Ankush Kumar, et al.
Published: (2025)

Biochemical Prostate Cancer Recurrence Prediction: Thinking Fast & Slow
by: You, Suhang, et al.
Published: (2024)

The Cutting Edge Quest: Epic Saga of Carboxymethyl Guar Gum in Drug Delivery and Roads Ahead
by: Vaishali Thakur, et al.
Published: (2025)

Alternatively Connected Corrole‐Porphyrin Multimetallic Complexes: Synthesis, Structure, and Properties
by: Harpal, et al.
Published: (2024)

A comprehensive review of nano clay: From development and applications to research opportunities
by: Piyush Gupta, et al.
Published: (2025)

One‐Step Nanoimprinting of Fe2O3/AgBr Thin Films for Dark‐Light Active Antibiofilm and Bacterial‐Free Cell Culture Surfaces
by: Anindita Thakur, et al.
Published: (2024)

Taylor spectrum of a Banach module over the quantum plane
by: Dosi, Anar
Published: (2024)

Noncommutative complex analytic geometry of a contractive quantum plane
by: Dosi, Anar
Published: (2024)

La economia del cambio técnico y el comercio internacional / Giovanni Dosi, Keith Pavitt, Luc Soete ; traducción de Maria de la Paz Mayo García
by: Dosi, Giovanni

Noncommutative localizations of a contractive quantum plane
by: Dosi, Anar
Published: (2025)

Homological framework of noncommutative complex analytic geometry and functional calculus
by: Dosi, Anar
Published: (2025)

Exploring the Age of Artificial Intelligence
by: Thakur, Mehak
Published: (2025)

AutoTrain: No-code training for state-of-the-art models
by: Thakur, Abhishek
Published: (2024)

Culturally-Grounded Chain-of-Thought (CG-CoT):Enhancing LLM Performance on Culturally-Specific Tasks in Low-Resource Languages
by: Thakur, Madhavendra
Published: (2025)

Variational Approximated Restricted Maximum Likelihood Estimation for Spatial Data
by: Thakur, Debjoy
Published: (2026)

Towards Neural No-Resource Language Translation: A Comparative Evaluation of Approaches
by: Thakur, Madhavendra
Published: (2024)

A Subsampling Based Neural Network for Spatial Data
by: Thakur, Debjoy
Published: (2024)

Feasibility of Dark Matter in Neutron Stars: A Quantitative Analysis
by: Thakur, Prashant
Published: (2025)

Integrating Zero-Shot Classification to Advance Long COVID Literature: A Systematic Social Media-Centered Review
by: Thakur, Nirmalya
Published: (2024)

A note on a subclass of bazilevič functions
by: Thakur, Lokenath
Published: (2026)

A Mixed-Gauge Caratheodory Measure Bridging Lebesgue Volume and Surface Content
by: Thakur, Yash
Published: (2025)

Not‐so‐Freeway: A Relational Approach to Checkpoints and Conflict in Northeast India
by: Shalaka Thakur
Published: (2026)