Cover Image

Performance Evaluation of Tokenizers in Large Language Models for the Assamese Language

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Tamang, Sagar, Bora, Dibya Jyoti
Format:	Preprint
Published:	2024
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Evaluating Tokenizer Performance of Large Language Models Across Official Indian Languages
by: Tamang, S., et al.
Published: (2024)

Enhancing Assamese NLP Capabilities: Introducing a Centralized Dataset Repository
by: Tamang, S., et al.
Published: (2024)

Enforcement Agents: Enhancing Accountability and Resilience in Multi-Agent AI Frameworks
by: Tamang, Sagar, et al.
Published: (2025)

Comparative Evaluation of Hard and Soft Clustering for Precise Brain Tumor Segmentation in MR Imaging
by: Bora, Dibya Jyoti, et al.
Published: (2025)

AC-Lite : A Lightweight Image Captioning Model for Low-Resource Assamese Language
by: Choudhury, Pankaj, et al.
Published: (2025)

Identifying and Analyzing Performance-Critical Tokens in Large Language Models
by: Bai, Yu, et al.
Published: (2024)

Enhancing rice leaf images: An overview of image denoising techniques
by: Chutia, Rupjyoti, et al.
Published: (2025)

Problematic Tokens: Tokenizer Bias in Large Language Models
by: Yang, Jin, et al.
Published: (2024)

Rethinking Tokenization: Crafting Better Tokenizers for Large Language Models
by: Yang, Jinbiao
Published: (2024)

Retrofitting Large Language Models with Dynamic Tokenization
by: Feher, Darius, et al.
Published: (2024)

Large Language Model as Token Compressor and Decompressor
by: Li, Wenbing, et al.
Published: (2026)

Deciphering Assamese Vowel Harmony with Featural InfoWaveGAN
by: Barman, Sneha Ray, et al.
Published: (2024)

Evaluation of Large Language Models via Coupled Token Generation
by: Benz, Nina Corvelo, et al.
Published: (2025)

Rethinking Personalization in Large Language Models at the Token Level
by: Zhang, Chenheng, et al.
Published: (2026)

Token-Level Privacy in Large Language Models
by: Harel, Re'em, et al.
Published: (2025)

Taking a Deep Breath: Enhancing Language Modeling of Large Language Models with Sentinel Tokens
by: Luo, Weiyao, et al.
Published: (2024)

Evaluating the Performance of Large Language Models on GAOKAO Benchmark
by: Zhang, Xiaotian, et al.
Published: (2023)

TASE: Token Awareness and Structured Evaluation for Multilingual Language Models
by: Zhao, Chenzhuo, et al.
Published: (2025)

Tokenization Matters! Degrading Large Language Models through Challenging Their Tokenization
by: Wang, Dixuan, et al.
Published: (2024)

Revisiting Graph-Tokenizing Large Language Models: A Systematic Evaluation of Graph Token Understanding
by: Zhang, Zhongjian, et al.
Published: (2026)

Tokenization Falling Short: On Subword Robustness in Large Language Models
by: Chai, Yekun, et al.
Published: (2024)

Contextually Structured Token Dependency Encoding for Large Language Models
by: Blades, James, et al.
Published: (2025)

MorphPiece : A Linguistic Tokenizer for Large Language Models
by: Jabbar, Haris
Published: (2023)

Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET Benchmark
by: Choi, Minje, et al.
Published: (2023)

Vision-centric Token Compression in Large Language Model
by: Xing, Ling, et al.
Published: (2025)

Evaluating Morphological Alignment of Tokenizers in 70 Languages
by: Arnett, Catherine, et al.
Published: (2025)

Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models
by: Levy, Mosh, et al.
Published: (2024)

Counting Ability of Large Language Models and Impact of Tokenization
by: Zhang, Xiang, et al.
Published: (2024)

The Geometry of Tokens in Internal Representations of Large Language Models
by: Viswanathan, Karthik, et al.
Published: (2025)

LBPE: Long-token-first Tokenization to Improve Large Language Models
by: Lian, Haoran, et al.
Published: (2024)

Flexibly Scaling Large Language Models Contexts Through Extensible Tokenization
by: Shao, Ninglu, et al.
Published: (2024)

Addressing Tokenization Inconsistency in Steganography and Watermarking Based on Large Language Models
by: Yan, Ruiyi, et al.
Published: (2025)

Structured Token Retention and Computational Memory Paths in Large Language Models
by: Delena, Jonathan, et al.
Published: (2025)

STARS: Synchronous Token Alignment for Robust Supervision in Large Language Models
by: Quamar, Mohammad Atif, et al.
Published: (2025)

TokMem: One-Token Procedural Memory for Large Language Models
by: Wu, Zijun, et al.
Published: (2025)

Evaluating the Impact of Compression Techniques on Task-Specific Performance of Large Language Models
by: Khanal, Bishwash, et al.
Published: (2024)

Performance Evaluation of Large Language Models in Bangla Consumer Health Query Summarization
by: Abrar, Ajwad, et al.
Published: (2025)

Towards Linguistically-Aware and Language-Independent Tokenization for Large Language Models (LLMs)
by: Rahman, Abrar, et al.
Published: (2024)

FTP: A Fine-grained Token-wise Pruner for Large Language Models via Token Routing
by: Li, Zekai, et al.
Published: (2024)

Counterfactual Token Generation in Large Language Models
by: Chatzi, Ivi, et al.
Published: (2024)