Copertina

Performance Evaluation of Tokenizers in Large Language Models for the Assamese Language

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Tamang, Sagar, Bora, Dibya Jyoti
Natura:	Preprint
Pubblicazione:	2024
Soggetti:	Computation and Language
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Evaluating Tokenizer Performance of Large Language Models Across Official Indian Languages
di: Tamang, S., et al.
Pubblicazione: (2024)

Enhancing Assamese NLP Capabilities: Introducing a Centralized Dataset Repository
di: Tamang, S., et al.
Pubblicazione: (2024)

Enforcement Agents: Enhancing Accountability and Resilience in Multi-Agent AI Frameworks
di: Tamang, Sagar, et al.
Pubblicazione: (2025)

Comparative Evaluation of Hard and Soft Clustering for Precise Brain Tumor Segmentation in MR Imaging
di: Bora, Dibya Jyoti, et al.
Pubblicazione: (2025)

AC-Lite : A Lightweight Image Captioning Model for Low-Resource Assamese Language
di: Choudhury, Pankaj, et al.
Pubblicazione: (2025)

Identifying and Analyzing Performance-Critical Tokens in Large Language Models
di: Bai, Yu, et al.
Pubblicazione: (2024)

Enhancing rice leaf images: An overview of image denoising techniques
di: Chutia, Rupjyoti, et al.
Pubblicazione: (2025)

Problematic Tokens: Tokenizer Bias in Large Language Models
di: Yang, Jin, et al.
Pubblicazione: (2024)

Rethinking Tokenization: Crafting Better Tokenizers for Large Language Models
di: Yang, Jinbiao
Pubblicazione: (2024)

Retrofitting Large Language Models with Dynamic Tokenization
di: Feher, Darius, et al.
Pubblicazione: (2024)

Large Language Model as Token Compressor and Decompressor
di: Li, Wenbing, et al.
Pubblicazione: (2026)

Deciphering Assamese Vowel Harmony with Featural InfoWaveGAN
di: Barman, Sneha Ray, et al.
Pubblicazione: (2024)

Evaluation of Large Language Models via Coupled Token Generation
di: Benz, Nina Corvelo, et al.
Pubblicazione: (2025)

Rethinking Personalization in Large Language Models at the Token Level
di: Zhang, Chenheng, et al.
Pubblicazione: (2026)

Token-Level Privacy in Large Language Models
di: Harel, Re'em, et al.
Pubblicazione: (2025)

Taking a Deep Breath: Enhancing Language Modeling of Large Language Models with Sentinel Tokens
di: Luo, Weiyao, et al.
Pubblicazione: (2024)

Evaluating the Performance of Large Language Models on GAOKAO Benchmark
di: Zhang, Xiaotian, et al.
Pubblicazione: (2023)

TASE: Token Awareness and Structured Evaluation for Multilingual Language Models
di: Zhao, Chenzhuo, et al.
Pubblicazione: (2025)

Tokenization Matters! Degrading Large Language Models through Challenging Their Tokenization
di: Wang, Dixuan, et al.
Pubblicazione: (2024)

Revisiting Graph-Tokenizing Large Language Models: A Systematic Evaluation of Graph Token Understanding
di: Zhang, Zhongjian, et al.
Pubblicazione: (2026)

Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET Benchmark
di: Choi, Minje, et al.
Pubblicazione: (2023)

Tokenization Falling Short: On Subword Robustness in Large Language Models
di: Chai, Yekun, et al.
Pubblicazione: (2024)

Contextually Structured Token Dependency Encoding for Large Language Models
di: Blades, James, et al.
Pubblicazione: (2025)

MorphPiece : A Linguistic Tokenizer for Large Language Models
di: Jabbar, Haris
Pubblicazione: (2023)

Vision-centric Token Compression in Large Language Model
di: Xing, Ling, et al.
Pubblicazione: (2025)

Evaluating Morphological Alignment of Tokenizers in 70 Languages
di: Arnett, Catherine, et al.
Pubblicazione: (2025)

Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models
di: Levy, Mosh, et al.
Pubblicazione: (2024)

Counting Ability of Large Language Models and Impact of Tokenization
di: Zhang, Xiang, et al.
Pubblicazione: (2024)

The Geometry of Tokens in Internal Representations of Large Language Models
di: Viswanathan, Karthik, et al.
Pubblicazione: (2025)

LBPE: Long-token-first Tokenization to Improve Large Language Models
di: Lian, Haoran, et al.
Pubblicazione: (2024)

Flexibly Scaling Large Language Models Contexts Through Extensible Tokenization
di: Shao, Ninglu, et al.
Pubblicazione: (2024)

Addressing Tokenization Inconsistency in Steganography and Watermarking Based on Large Language Models
di: Yan, Ruiyi, et al.
Pubblicazione: (2025)

Structured Token Retention and Computational Memory Paths in Large Language Models
di: Delena, Jonathan, et al.
Pubblicazione: (2025)

STARS: Synchronous Token Alignment for Robust Supervision in Large Language Models
di: Quamar, Mohammad Atif, et al.
Pubblicazione: (2025)

TokMem: One-Token Procedural Memory for Large Language Models
di: Wu, Zijun, et al.
Pubblicazione: (2025)

Evaluating the Impact of Compression Techniques on Task-Specific Performance of Large Language Models
di: Khanal, Bishwash, et al.
Pubblicazione: (2024)

Performance Evaluation of Large Language Models in Bangla Consumer Health Query Summarization
di: Abrar, Ajwad, et al.
Pubblicazione: (2025)

Towards Linguistically-Aware and Language-Independent Tokenization for Large Language Models (LLMs)
di: Rahman, Abrar, et al.
Pubblicazione: (2024)

FTP: A Fine-grained Token-wise Pruner for Large Language Models via Token Routing
di: Li, Zekai, et al.
Pubblicazione: (2024)

Counterfactual Token Generation in Large Language Models
di: Chatzi, Ivi, et al.
Pubblicazione: (2024)