Image de couverture de livre

Evaluating Tokenizer Performance of Large Language Models Across Official Indian Languages

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Tamang, S., Bora, D. J.
Format:	Preprint
Publié:	2024
Sujets:	Computation and Language Artificial Intelligence
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Performance Evaluation of Tokenizers in Large Language Models for the Assamese Language
par: Tamang, Sagar, et autres
Publié: (2024)

Enhancing Assamese NLP Capabilities: Introducing a Centralized Dataset Repository
par: Tamang, S., et autres
Publié: (2024)

Evaluating the Performance of Large Language Models on GAOKAO Benchmark
par: Zhang, Xiaotian, et autres
Publié: (2023)

Tokenization Matters! Degrading Large Language Models through Challenging Their Tokenization
par: Wang, Dixuan, et autres
Publié: (2024)

An Innovative Solution: AI-Based Digital Screen-Integrated Tables for Educational Settings
par: Tamang, S., et autres
Publié: (2024)

Evaluation of Large Language Models via Coupled Token Generation
par: Benz, Nina Corvelo, et autres
Publié: (2025)

IndicEval: A Bilingual Indian Educational Evaluation Framework for Large Language Models
par: Bharti, Saurabh, et autres
Publié: (2026)

Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models
par: Levy, Mosh, et autres
Publié: (2024)

Counting Ability of Large Language Models and Impact of Tokenization
par: Zhang, Xiang, et autres
Publié: (2024)

Revisiting Graph-Tokenizing Large Language Models: A Systematic Evaluation of Graph Token Understanding
par: Zhang, Zhongjian, et autres
Publié: (2026)

Evaluating Open-Weight Large Language Models for Structured Data Extraction from Narrative Medical Reports Across Multiple Use Cases and Languages
par: Spaanderman, Douwe J., et autres
Publié: (2025)

On the Fallacy of Global Token Perplexity in Spoken Language Model Evaluation
par: Hsu, Chan-Jan, et autres
Publié: (2026)

Contextual Reinforcement in Multimodal Token Compression for Large Language Models
par: Piero, Naderdel, et autres
Publié: (2025)

Saliency-driven Dynamic Token Pruning for Large Language Models
par: Tao, Yao, et autres
Publié: (2025)

Is There a Case for Conversation Optimized Tokenizers in Large Language Models?
par: Ferrando, Raquel, et autres
Publié: (2025)

A Peek into Token Bias: Large Language Models Are Not Yet Genuine Reasoners
par: Jiang, Bowen, et autres
Publié: (2024)

Evaluating the Translation Performance of Large Language Models Based on Euas-20
par: Huang, Yan, et autres
Publié: (2024)

LoASR-Bench: Evaluating Large Speech Language Models on Low-Resource Automatic Speech Recognition Across Language Families
par: Chen, Jianan, et autres
Publié: (2026)

How Much Data is Enough Data? Fine-Tuning Large Language Models for In-House Translation: Performance Evaluation Across Multiple Dataset Sizes
par: Vieira, Inacio, et autres
Publié: (2024)

MuBench: Assessment of Multilingual Capabilities of Large Language Models Across 61 Languages
par: Han, Wenhan, et autres
Publié: (2025)

Token Signature: Predicting Chain-of-Thought Gains with Token Decoding Feature in Large Language Models
par: Liu, Peijie, et autres
Publié: (2025)

Thinking Tokens for Language Modeling
par: Herel, David, et autres
Publié: (2024)

From Language Models over Tokens to Language Models over Characters
par: Vieira, Tim, et autres
Publié: (2024)

Do Language Models Reason Across Languages?
par: Meng, Yan, et autres
Publié: (2026)

UNITYAI-GUARD: Pioneering Toxicity Detection Across Low-Resource Indian Languages
par: Beniwal, Himanshu, et autres
Publié: (2025)

Re-Initialization Token Learning for Tool-Augmented Large Language Models
par: Li, Chenghao, et autres
Publié: (2025)

Memory Retrieval and Consolidation in Large Language Models through Function Tokens
par: Zhang, Shaohua, et autres
Publié: (2025)

TEAL: Tokenize and Embed ALL for Multi-modal Large Language Models
par: Yang, Zhen, et autres
Publié: (2023)

Enhancing Large Language Models for Mobility Analytics with Semantic Location Tokenization
par: Chen, Yile, et autres
Publié: (2025)

Systematic Weight Evaluation for Pruning Large Language Models: Enhancing Performance and Sustainability
par: Islam, Ashhadul, et autres
Publié: (2025)

Far Out: Evaluating Language Models on Slang in Australian and Indian English
par: Dilsiz, Deniz Kaya, et autres
Publié: (2026)

Counterfactual Token Generation in Large Language Models
par: Chatzi, Ivi, et autres
Publié: (2024)

Cultural Fidelity in Large-Language Models: An Evaluation of Online Language Resources as a Driver of Model Performance in Value Representation
par: Kazemi, Sharif, et autres
Publié: (2024)

Large Language Models Meet Contrastive Learning: Zero-Shot Emotion Recognition Across Languages
par: Zou, Heqing, et autres
Publié: (2025)

A Law of Next-Token Prediction in Large Language Models
par: He, Hangfeng, et autres
Publié: (2024)

Accelerating Multilingual Language Model for Excessively Tokenized Languages
par: Hong, Jimin, et autres
Publié: (2024)

Evaluating the Performance of Large Language Models via Debates
par: Moniri, Behrad, et autres
Publié: (2024)

Evaluating Modern Large Language Models on Low-Resource and Morphologically Rich Languages:A Cross-Lingual Benchmark Across Cantonese, Japanese, and Turkish
par: Xia, Chengxuan, et autres
Publié: (2025)

Evaluating Quantized Large Language Models
par: Li, Shiyao, et autres
Publié: (2024)

Unveiling Selection Biases: Exploring Order and Token Sensitivity in Large Language Models
par: Wei, Sheng-Lun, et autres
Publié: (2024)