Image de couverture de livre

Performance Evaluation of Tokenizers in Large Language Models for the Assamese Language

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Tamang, Sagar, Bora, Dibya Jyoti
Format:	Preprint
Publié:	2024
Sujets:	Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Evaluating Tokenizer Performance of Large Language Models Across Official Indian Languages
par: Tamang, S., et autres
Publié: (2024)

Enhancing Assamese NLP Capabilities: Introducing a Centralized Dataset Repository
par: Tamang, S., et autres
Publié: (2024)

Enforcement Agents: Enhancing Accountability and Resilience in Multi-Agent AI Frameworks
par: Tamang, Sagar, et autres
Publié: (2025)

Comparative Evaluation of Hard and Soft Clustering for Precise Brain Tumor Segmentation in MR Imaging
par: Bora, Dibya Jyoti, et autres
Publié: (2025)

AC-Lite : A Lightweight Image Captioning Model for Low-Resource Assamese Language
par: Choudhury, Pankaj, et autres
Publié: (2025)

Identifying and Analyzing Performance-Critical Tokens in Large Language Models
par: Bai, Yu, et autres
Publié: (2024)

Enhancing rice leaf images: An overview of image denoising techniques
par: Chutia, Rupjyoti, et autres
Publié: (2025)

Problematic Tokens: Tokenizer Bias in Large Language Models
par: Yang, Jin, et autres
Publié: (2024)

Rethinking Tokenization: Crafting Better Tokenizers for Large Language Models
par: Yang, Jinbiao
Publié: (2024)

Retrofitting Large Language Models with Dynamic Tokenization
par: Feher, Darius, et autres
Publié: (2024)

Large Language Model as Token Compressor and Decompressor
par: Li, Wenbing, et autres
Publié: (2026)

Deciphering Assamese Vowel Harmony with Featural InfoWaveGAN
par: Barman, Sneha Ray, et autres
Publié: (2024)

Evaluation of Large Language Models via Coupled Token Generation
par: Benz, Nina Corvelo, et autres
Publié: (2025)

Rethinking Personalization in Large Language Models at the Token Level
par: Zhang, Chenheng, et autres
Publié: (2026)

Token-Level Privacy in Large Language Models
par: Harel, Re'em, et autres
Publié: (2025)

Taking a Deep Breath: Enhancing Language Modeling of Large Language Models with Sentinel Tokens
par: Luo, Weiyao, et autres
Publié: (2024)

Evaluating the Performance of Large Language Models on GAOKAO Benchmark
par: Zhang, Xiaotian, et autres
Publié: (2023)

TASE: Token Awareness and Structured Evaluation for Multilingual Language Models
par: Zhao, Chenzhuo, et autres
Publié: (2025)

Tokenization Matters! Degrading Large Language Models through Challenging Their Tokenization
par: Wang, Dixuan, et autres
Publié: (2024)

Revisiting Graph-Tokenizing Large Language Models: A Systematic Evaluation of Graph Token Understanding
par: Zhang, Zhongjian, et autres
Publié: (2026)

Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET Benchmark
par: Choi, Minje, et autres
Publié: (2023)

Tokenization Falling Short: On Subword Robustness in Large Language Models
par: Chai, Yekun, et autres
Publié: (2024)

Contextually Structured Token Dependency Encoding for Large Language Models
par: Blades, James, et autres
Publié: (2025)

MorphPiece : A Linguistic Tokenizer for Large Language Models
par: Jabbar, Haris
Publié: (2023)

Vision-centric Token Compression in Large Language Model
par: Xing, Ling, et autres
Publié: (2025)

Evaluating Morphological Alignment of Tokenizers in 70 Languages
par: Arnett, Catherine, et autres
Publié: (2025)

Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models
par: Levy, Mosh, et autres
Publié: (2024)

Counting Ability of Large Language Models and Impact of Tokenization
par: Zhang, Xiang, et autres
Publié: (2024)

The Geometry of Tokens in Internal Representations of Large Language Models
par: Viswanathan, Karthik, et autres
Publié: (2025)

LBPE: Long-token-first Tokenization to Improve Large Language Models
par: Lian, Haoran, et autres
Publié: (2024)

Flexibly Scaling Large Language Models Contexts Through Extensible Tokenization
par: Shao, Ninglu, et autres
Publié: (2024)

Addressing Tokenization Inconsistency in Steganography and Watermarking Based on Large Language Models
par: Yan, Ruiyi, et autres
Publié: (2025)

Structured Token Retention and Computational Memory Paths in Large Language Models
par: Delena, Jonathan, et autres
Publié: (2025)

STARS: Synchronous Token Alignment for Robust Supervision in Large Language Models
par: Quamar, Mohammad Atif, et autres
Publié: (2025)

TokMem: One-Token Procedural Memory for Large Language Models
par: Wu, Zijun, et autres
Publié: (2025)

Evaluating the Impact of Compression Techniques on Task-Specific Performance of Large Language Models
par: Khanal, Bishwash, et autres
Publié: (2024)

Performance Evaluation of Large Language Models in Bangla Consumer Health Query Summarization
par: Abrar, Ajwad, et autres
Publié: (2025)

Towards Linguistically-Aware and Language-Independent Tokenization for Large Language Models (LLMs)
par: Rahman, Abrar, et autres
Publié: (2024)

FTP: A Fine-grained Token-wise Pruner for Large Language Models via Token Routing
par: Li, Zekai, et autres
Publié: (2024)

Counterfactual Token Generation in Large Language Models
par: Chatzi, Ivi, et autres
Publié: (2024)