Copertina

Evaluating Tokenizer Performance of Large Language Models Across Official Indian Languages

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Tamang, S., Bora, D. J.
Natura:	Preprint
Pubblicazione:	2024
Soggetti:	Computation and Language Artificial Intelligence
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Performance Evaluation of Tokenizers in Large Language Models for the Assamese Language
di: Tamang, Sagar, et al.
Pubblicazione: (2024)

Enhancing Assamese NLP Capabilities: Introducing a Centralized Dataset Repository
di: Tamang, S., et al.
Pubblicazione: (2024)

Evaluating the Performance of Large Language Models on GAOKAO Benchmark
di: Zhang, Xiaotian, et al.
Pubblicazione: (2023)

Tokenization Matters! Degrading Large Language Models through Challenging Their Tokenization
di: Wang, Dixuan, et al.
Pubblicazione: (2024)

An Innovative Solution: AI-Based Digital Screen-Integrated Tables for Educational Settings
di: Tamang, S., et al.
Pubblicazione: (2024)

Evaluation of Large Language Models via Coupled Token Generation
di: Benz, Nina Corvelo, et al.
Pubblicazione: (2025)

IndicEval: A Bilingual Indian Educational Evaluation Framework for Large Language Models
di: Bharti, Saurabh, et al.
Pubblicazione: (2026)

Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models
di: Levy, Mosh, et al.
Pubblicazione: (2024)

Counting Ability of Large Language Models and Impact of Tokenization
di: Zhang, Xiang, et al.
Pubblicazione: (2024)

Revisiting Graph-Tokenizing Large Language Models: A Systematic Evaluation of Graph Token Understanding
di: Zhang, Zhongjian, et al.
Pubblicazione: (2026)

Evaluating Open-Weight Large Language Models for Structured Data Extraction from Narrative Medical Reports Across Multiple Use Cases and Languages
di: Spaanderman, Douwe J., et al.
Pubblicazione: (2025)

On the Fallacy of Global Token Perplexity in Spoken Language Model Evaluation
di: Hsu, Chan-Jan, et al.
Pubblicazione: (2026)

Contextual Reinforcement in Multimodal Token Compression for Large Language Models
di: Piero, Naderdel, et al.
Pubblicazione: (2025)

Saliency-driven Dynamic Token Pruning for Large Language Models
di: Tao, Yao, et al.
Pubblicazione: (2025)

Is There a Case for Conversation Optimized Tokenizers in Large Language Models?
di: Ferrando, Raquel, et al.
Pubblicazione: (2025)

A Peek into Token Bias: Large Language Models Are Not Yet Genuine Reasoners
di: Jiang, Bowen, et al.
Pubblicazione: (2024)

Evaluating the Translation Performance of Large Language Models Based on Euas-20
di: Huang, Yan, et al.
Pubblicazione: (2024)

LoASR-Bench: Evaluating Large Speech Language Models on Low-Resource Automatic Speech Recognition Across Language Families
di: Chen, Jianan, et al.
Pubblicazione: (2026)

How Much Data is Enough Data? Fine-Tuning Large Language Models for In-House Translation: Performance Evaluation Across Multiple Dataset Sizes
di: Vieira, Inacio, et al.
Pubblicazione: (2024)

MuBench: Assessment of Multilingual Capabilities of Large Language Models Across 61 Languages
di: Han, Wenhan, et al.
Pubblicazione: (2025)

Token Signature: Predicting Chain-of-Thought Gains with Token Decoding Feature in Large Language Models
di: Liu, Peijie, et al.
Pubblicazione: (2025)

Thinking Tokens for Language Modeling
di: Herel, David, et al.
Pubblicazione: (2024)

From Language Models over Tokens to Language Models over Characters
di: Vieira, Tim, et al.
Pubblicazione: (2024)

Do Language Models Reason Across Languages?
di: Meng, Yan, et al.
Pubblicazione: (2026)

UNITYAI-GUARD: Pioneering Toxicity Detection Across Low-Resource Indian Languages
di: Beniwal, Himanshu, et al.
Pubblicazione: (2025)

Re-Initialization Token Learning for Tool-Augmented Large Language Models
di: Li, Chenghao, et al.
Pubblicazione: (2025)

Memory Retrieval and Consolidation in Large Language Models through Function Tokens
di: Zhang, Shaohua, et al.
Pubblicazione: (2025)

TEAL: Tokenize and Embed ALL for Multi-modal Large Language Models
di: Yang, Zhen, et al.
Pubblicazione: (2023)

Enhancing Large Language Models for Mobility Analytics with Semantic Location Tokenization
di: Chen, Yile, et al.
Pubblicazione: (2025)

Systematic Weight Evaluation for Pruning Large Language Models: Enhancing Performance and Sustainability
di: Islam, Ashhadul, et al.
Pubblicazione: (2025)

Far Out: Evaluating Language Models on Slang in Australian and Indian English
di: Dilsiz, Deniz Kaya, et al.
Pubblicazione: (2026)

Counterfactual Token Generation in Large Language Models
di: Chatzi, Ivi, et al.
Pubblicazione: (2024)

Cultural Fidelity in Large-Language Models: An Evaluation of Online Language Resources as a Driver of Model Performance in Value Representation
di: Kazemi, Sharif, et al.
Pubblicazione: (2024)

Large Language Models Meet Contrastive Learning: Zero-Shot Emotion Recognition Across Languages
di: Zou, Heqing, et al.
Pubblicazione: (2025)

A Law of Next-Token Prediction in Large Language Models
di: He, Hangfeng, et al.
Pubblicazione: (2024)

Accelerating Multilingual Language Model for Excessively Tokenized Languages
di: Hong, Jimin, et al.
Pubblicazione: (2024)

Evaluating the Performance of Large Language Models via Debates
di: Moniri, Behrad, et al.
Pubblicazione: (2024)

Evaluating Modern Large Language Models on Low-Resource and Morphologically Rich Languages:A Cross-Lingual Benchmark Across Cantonese, Japanese, and Turkish
di: Xia, Chengxuan, et al.
Pubblicazione: (2025)

Evaluating Quantized Large Language Models
di: Li, Shiyao, et al.
Pubblicazione: (2024)

Unveiling Selection Biases: Exploring Order and Token Sensitivity in Large Language Models
di: Wei, Sheng-Lun, et al.
Pubblicazione: (2024)