Imagen de Portada

MUTANT: A Recipe for Multilingual Tokenizer Design

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Rana, Souvik, Menezes, Arul, Kulkarni, Ashish, Khatri, Chandra, Agarwal, Shubham
Formato:	Preprint
Publicado:	2025
Materias:	Computation and Language
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

BhashaKritika: Building Synthetic Pretraining Data at Scale for Indic Languages
por: Manoj, Guduru, et al.
Publicado: (2025)

Pragyaan: Designing and Curating High-Quality Cultural Post-Training Datasets for Indian Languages
por: Rachamalla, Neel Prabhanjan, et al.
Publicado: (2025)

VoiceAgentBench: Are Voice Assistants ready for agentic tasks?
por: Jain, Dhruv, et al.
Publicado: (2025)

Krutrim LLM: A Novel Tokenization Strategy for Multilingual Indic Languages with Petabyte-Scale Data Processing
por: Kumar, Rahul, et al.
Publicado: (2024)

Chitrakshara: A Large Multilingual Multimodal Dataset for Indian languages
por: Khan, Shaharukh, et al.
Publicado: (2026)

Chitranuvad: Adapting Multi-Lingual LLMs for Multimodal Translation
por: Khan, Shaharukh, et al.
Publicado: (2025)

Designing Production-Scale OCR for India: Multilingual and Domain-Specific Systems
por: Faraz, Ali, et al.
Publicado: (2026)

Krutrim LLM: Multilingual Foundational Model for over a Billion People
por: Kallappa, Aditya, et al.
Publicado: (2025)

Seeing Straight: Document Orientation Detection for Efficient OCR
por: Goswami, Suranjan, et al.
Publicado: (2025)

The Digital Sous Chef -- A Comparative Study on Fine-Tuning Language Models for Recipe Generation
por: Pundhir, Shubham, et al.
Publicado: (2025)

A Recipe of Parallel Corpora Exploitation for Multilingual Large Language Models
por: Lin, Peiqin, et al.
Publicado: (2024)

Deep Learning Based Named Entity Recognition Models for Recipes
por: Goel, Mansi, et al.
Publicado: (2024)

A Design Recipe and Recipe-Based Errors for Regular Expressions
por: Morazán, Marco T., et al.
Publicado: (2025)

The Art of Breaking Words: Rethinking Multilingual Tokenizer Design
por: Thakur, Aamod, et al.
Publicado: (2025)

Beyond Literal Token Overlap: Token Alignability for Multilinguality
por: Hämmerl, Katharina, et al.
Publicado: (2025)

The Token Tax: Systematic Bias in Multilingual Tokenization
por: Lundin, Jessica M., et al.
Publicado: (2025)

EXECUTE: A Multilingual Benchmark for LLM Token Understanding
por: Edman, Lukas, et al.
Publicado: (2025)

Chitrarth: Bridging Vision and Language for a Billion People
por: Khan, Shaharukh, et al.
Publicado: (2025)

A Survey of Multilingual Reasoning in Language Models
por: Ghosh, Akash, et al.
Publicado: (2025)

INSURE-Dial: A Phase-Aware Conversational Dataset & Benchmark for Compliance Verification and Phase Detection
por: Kulkarni, Shubham, et al.
Publicado: (2026)

MATHSENSEI: A Tool-Augmented Large Language Model for Mathematical Reasoning
por: Das, Debrup, et al.
Publicado: (2024)

Towards Automatic Evaluation of Task-Oriented Dialogue Flows
por: Mirtaheri, Mehrnoosh, et al.
Publicado: (2024)

One Tokenizer To Rule Them All: Emergent Language Plasticity via Multilingual Tokenizers
por: Abagyan, Diana, et al.
Publicado: (2025)

LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Models
por: Hu, Zhiyuan, et al.
Publicado: (2024)

GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM
por: Kang, Hao, et al.
Publicado: (2024)

A U-Net and Transformer Pipeline for Multilingual Image Translation
por: Sahay, Siddharth, et al.
Publicado: (2025)

How Language Directions Align with Token Geometry in Multilingual LLMs
por: Kim, JaeSeong, et al.
Publicado: (2025)

TASE: Token Awareness and Structured Evaluation for Multilingual Language Models
por: Zhao, Chenzhuo, et al.
Publicado: (2025)

Multilingual Tokenization through the Lens of Indian Languages: Challenges and Insights
por: Karthika, N J, et al.
Publicado: (2025)

Dynamic Multi-Expert Projectors with Stabilized Routing for Multilingual Speech Recognition
por: Pandey, Isha, et al.
Publicado: (2026)

IndicVisionBench: Benchmarking Cultural and Multilingual Understanding in VLMs
por: Faraz, Ali, et al.
Publicado: (2025)

Token-free Models for Sarcasm Detection
por: Mamtani, Sumit, et al.
Publicado: (2025)

Improving Multilingual Language Models by Aligning Representations through Steering
por: Mahmoud, Omar, et al.
Publicado: (2025)

ReGATE: Learning Faster and Better with Fewer Tokens in MLLMs
por: Li, Chaoyu, et al.
Publicado: (2025)

Ratatouille: A tool for Novel Recipe Generation
por: Goel, Mansi, et al.
Publicado: (2022)

Multilingual Hidden Prompt Injection Attacks on LLM-Based Academic Reviewing
por: Theocharopoulos, Panagiotis, et al.
Publicado: (2025)

One Token Away from Collapse: The Fragility of Instruction-Tuned Helpfulness
por: Potraghloo, Erfan Baghaei, et al.
Publicado: (2026)

Accelerating Multilingual Language Model for Excessively Tokenized Languages
por: Hong, Jimin, et al.
Publicado: (2024)

KULCQ: An Unsupervised Keyword-based Utterance Level Clustering Quality Metric
por: Guruprasad, Pranav, et al.
Publicado: (2024)

Signs as Tokens: A Retrieval-Enhanced Multilingual Sign Language Generator
por: Zuo, Ronglai, et al.
Publicado: (2024)