Imagen de Portada

Evaluating Tokenizer Performance of Large Language Models Across Official Indian Languages

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Tamang, S., Bora, D. J.
Formato:	Preprint
Publicado:	2024
Materias:	Computation and Language Artificial Intelligence
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Performance Evaluation of Tokenizers in Large Language Models for the Assamese Language
por: Tamang, Sagar, et al.
Publicado: (2024)

Enhancing Assamese NLP Capabilities: Introducing a Centralized Dataset Repository
por: Tamang, S., et al.
Publicado: (2024)

Evaluating the Performance of Large Language Models on GAOKAO Benchmark
por: Zhang, Xiaotian, et al.
Publicado: (2023)

Tokenization Matters! Degrading Large Language Models through Challenging Their Tokenization
por: Wang, Dixuan, et al.
Publicado: (2024)

An Innovative Solution: AI-Based Digital Screen-Integrated Tables for Educational Settings
por: Tamang, S., et al.
Publicado: (2024)

Evaluation of Large Language Models via Coupled Token Generation
por: Benz, Nina Corvelo, et al.
Publicado: (2025)

IndicEval: A Bilingual Indian Educational Evaluation Framework for Large Language Models
por: Bharti, Saurabh, et al.
Publicado: (2026)

Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models
por: Levy, Mosh, et al.
Publicado: (2024)

Counting Ability of Large Language Models and Impact of Tokenization
por: Zhang, Xiang, et al.
Publicado: (2024)

Revisiting Graph-Tokenizing Large Language Models: A Systematic Evaluation of Graph Token Understanding
por: Zhang, Zhongjian, et al.
Publicado: (2026)

Evaluating Open-Weight Large Language Models for Structured Data Extraction from Narrative Medical Reports Across Multiple Use Cases and Languages
por: Spaanderman, Douwe J., et al.
Publicado: (2025)

On the Fallacy of Global Token Perplexity in Spoken Language Model Evaluation
por: Hsu, Chan-Jan, et al.
Publicado: (2026)

Contextual Reinforcement in Multimodal Token Compression for Large Language Models
por: Piero, Naderdel, et al.
Publicado: (2025)

Saliency-driven Dynamic Token Pruning for Large Language Models
por: Tao, Yao, et al.
Publicado: (2025)

Is There a Case for Conversation Optimized Tokenizers in Large Language Models?
por: Ferrando, Raquel, et al.
Publicado: (2025)

A Peek into Token Bias: Large Language Models Are Not Yet Genuine Reasoners
por: Jiang, Bowen, et al.
Publicado: (2024)

Evaluating the Translation Performance of Large Language Models Based on Euas-20
por: Huang, Yan, et al.
Publicado: (2024)

LoASR-Bench: Evaluating Large Speech Language Models on Low-Resource Automatic Speech Recognition Across Language Families
por: Chen, Jianan, et al.
Publicado: (2026)

How Much Data is Enough Data? Fine-Tuning Large Language Models for In-House Translation: Performance Evaluation Across Multiple Dataset Sizes
por: Vieira, Inacio, et al.
Publicado: (2024)

MuBench: Assessment of Multilingual Capabilities of Large Language Models Across 61 Languages
por: Han, Wenhan, et al.
Publicado: (2025)

Token Signature: Predicting Chain-of-Thought Gains with Token Decoding Feature in Large Language Models
por: Liu, Peijie, et al.
Publicado: (2025)

Thinking Tokens for Language Modeling
por: Herel, David, et al.
Publicado: (2024)

From Language Models over Tokens to Language Models over Characters
por: Vieira, Tim, et al.
Publicado: (2024)

Do Language Models Reason Across Languages?
por: Meng, Yan, et al.
Publicado: (2026)

UNITYAI-GUARD: Pioneering Toxicity Detection Across Low-Resource Indian Languages
por: Beniwal, Himanshu, et al.
Publicado: (2025)

Re-Initialization Token Learning for Tool-Augmented Large Language Models
por: Li, Chenghao, et al.
Publicado: (2025)

Memory Retrieval and Consolidation in Large Language Models through Function Tokens
por: Zhang, Shaohua, et al.
Publicado: (2025)

TEAL: Tokenize and Embed ALL for Multi-modal Large Language Models
por: Yang, Zhen, et al.
Publicado: (2023)

Enhancing Large Language Models for Mobility Analytics with Semantic Location Tokenization
por: Chen, Yile, et al.
Publicado: (2025)

Systematic Weight Evaluation for Pruning Large Language Models: Enhancing Performance and Sustainability
por: Islam, Ashhadul, et al.
Publicado: (2025)

Far Out: Evaluating Language Models on Slang in Australian and Indian English
por: Dilsiz, Deniz Kaya, et al.
Publicado: (2026)

Counterfactual Token Generation in Large Language Models
por: Chatzi, Ivi, et al.
Publicado: (2024)

Cultural Fidelity in Large-Language Models: An Evaluation of Online Language Resources as a Driver of Model Performance in Value Representation
por: Kazemi, Sharif, et al.
Publicado: (2024)

Large Language Models Meet Contrastive Learning: Zero-Shot Emotion Recognition Across Languages
por: Zou, Heqing, et al.
Publicado: (2025)

A Law of Next-Token Prediction in Large Language Models
por: He, Hangfeng, et al.
Publicado: (2024)

Accelerating Multilingual Language Model for Excessively Tokenized Languages
por: Hong, Jimin, et al.
Publicado: (2024)

Evaluating the Performance of Large Language Models via Debates
por: Moniri, Behrad, et al.
Publicado: (2024)

Evaluating Modern Large Language Models on Low-Resource and Morphologically Rich Languages:A Cross-Lingual Benchmark Across Cantonese, Japanese, and Turkish
por: Xia, Chengxuan, et al.
Publicado: (2025)

Evaluating Quantized Large Language Models
por: Li, Shiyao, et al.
Publicado: (2024)

Unveiling Selection Biases: Exploring Order and Token Sensitivity in Large Language Models
por: Wei, Sheng-Lun, et al.
Publicado: (2024)