Image de couverture de livre

Comparative analysis of subword tokenization approaches for Indian languages

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Das, Sudhansu Bala, Choudhury, Samujjal, Mishra, Tapas Kumar, Patra, Bidyut Kr.
Format:	Preprint
Publié:	2025
Sujets:	Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

An approach for mistranslation removal from popular dataset for Indic MT Task
par: Das, Sudhansu Bala, et autres
Publié: (2024)

Statistical Machine Translation for Indic Languages
par: Das, Sudhansu Bala, et autres
Publié: (2023)

Improving Multilingual Neural Machine Translation System for Indic Languages
par: Das, Sudhansu Bala, et autres
Publié: (2022)

Revisiting subword tokenization: A case study on affixal negation in large language models
par: Truong, Thinh Hung, et autres
Publié: (2024)

AtteSTNet -- An attention and subword tokenization based approach for code-switched text hate speech detection
par: Shingi, Geet, et autres
Publié: (2021)

Morphological evaluation of subwords vocabulary used by BETO language model
par: García-Sierra, Óscar, et autres
Publié: (2024)

On state complexity for subword-closed languages
par: Guyot, Jérôme
Publié: (2024)

A Counterfactual Approach for Addressing Individual User Unfairness in Collaborative Recommender System
par: Baidya, Nikita, et autres
Publié: (2026)

Visualizing token importance for black-box language models
par: Rauba, Paulius, et autres
Publié: (2025)

Do language models plan ahead for future tokens?
par: Wu, Wilson, et autres
Publié: (2024)

DeepMLF: Multimodal language model with learnable tokens for deep fusion in sentiment analysis
par: Georgiou, Efthymios, et autres
Publié: (2025)

Collaborative decoding of critical tokens for boosting factuality of large language models
par: Jin, Lifeng, et autres
Publié: (2024)

Is Sanskrit the most token-efficient language? A quantitative study using GPT, Gemini, and SentencePiece
par: Kumar, Anshul
Publié: (2026)

On the scaling relationship between cloze probabilities and language model next-token prediction
par: Jacobs, Cassandra L., et autres
Publié: (2026)

Exploring News Summarization and Enrichment in a Highly Resource-Scarce Indian Language: A Case Study of Mizo
par: Bala, Abhinaba, et autres
Publié: (2024)

Interpreting token compositionality in LLMs: A robustness analysis
par: Aljaafari, Nura, et autres
Publié: (2024)

Towards Nepali-language LLMs: Efficient GPT training with a Nepali BPE tokenizer
par: Shrestha, Adarsha, et autres
Publié: (2025)

A cost minimization approach to fix the vocabulary size in a tokenizer for an End-to-End ASR system
par: Kopparapu, Sunil Kumar, et autres
Publié: (2024)

Multilingual Text Style Transfer: Datasets & Models for Indian Languages
par: Mukherjee, Sourabrata, et autres
Publié: (2024)

HalluCounter: Reference-free LLM Hallucination Detection in the Wild!
par: Urlana, Ashok, et autres
Publié: (2025)

Enhancing FKG.in: automating Indian food composition analysis
par: Gupta, Saransh Kumar, et autres
Publié: (2024)

Comparing LLM prompting with Cross-lingual transfer performance on Indigenous and Low-resource Brazilian Languages
par: Adelani, David Ifeoluwa, et autres
Publié: (2024)

IndicEval: A Bilingual Indian Educational Evaluation Framework for Large Language Models
par: Bharti, Saurabh, et autres
Publié: (2026)

CTG-KrEW: Generating Synthetic Structured Contextually Correlated Content by Conditional Tabular GAN with K-Means Clustering and Efficient Word Embedding
par: Samanta, Riya, et autres
Publié: (2024)

ILID: Native Script Language Identification for Indian Languages
par: Ingle, Yash, et autres
Publié: (2025)

Why do LLMs attend to the first token?
par: Barbero, Federico, et autres
Publié: (2025)

Where is the signal in tokenization space?
par: Geh, Renato Lui, et autres
Publié: (2024)

Arithmetical subword complexity of automatic sequences
par: Konieczny, Jakub, et autres
Publié: (2023)

LLMs with Industrial Lens: Deciphering the Challenges and Prospects -- A Survey
par: Urlana, Ashok, et autres
Publié: (2024)

Not a nuisance but a useful heuristic: Outlier dimensions favor frequent tokens in language models
par: Macocco, Iuri, et autres
Publié: (2025)

No LLM is Free From Bias: A Comprehensive Study of Bias Evaluation in Large Language Models
par: Kumar, Charaka Vinayak, et autres
Publié: (2025)

Contextual morphologically-guided tokenization for Latin encoder models
par: Hudspeth, Marisa, et autres
Publié: (2025)

SMAB: MAB based word Sensitivity Estimation Framework and its Applications in Adversarial Text Generation
par: Pandey, Saurabh Kumar, et autres
Publié: (2025)

AnomaLLMy -- Detecting anomalous tokens in black-box LLMs through low-confidence single-token predictions
par: Witold, Waligóra
Publié: (2024)

Semantically Cohesive Word Grouping in Indian Languages
par: Karthika, N J, et autres
Publié: (2025)

Looking beyond the next token
par: Thankaraj, Abitha, et autres
Publié: (2025)

Are Large Language Models Actually Good at Text Style Transfer?
par: Mukherjee, Sourabrata, et autres
Publié: (2024)

The pitfalls of next-token prediction
par: Bachmann, Gregor, et autres
Publié: (2024)

Efficient Aspect Term Extraction using Spiking Neural Network
par: Mishra, Abhishek Kumar, et autres
Publié: (2026)

On multi-token prediction for efficient LLM inference
par: Mehra, Somesh, et autres
Publié: (2025)