Copertina

Problematic Tokens: Tokenizer Bias in Large Language Models

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Yang, Jin, Wang, Zhiqiang, Lin, Yanbin, Zhao, Zunduo
Natura:	Preprint
Pubblicazione:	2024
Soggetti:	Computation and Language
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Benchmarking Large Language Models for Image Classification of Marine Mammals
di: Qi, Yijiashun, et al.
Pubblicazione: (2024)

Large Language Model as Token Compressor and Decompressor
di: Li, Wenbing, et al.
Pubblicazione: (2026)

Rethinking Tokenization: Crafting Better Tokenizers for Large Language Models
di: Yang, Jinbiao
Pubblicazione: (2024)

Adaptable and Reliable Text Classification using Large Language Models
di: Wang, Zhiqiang, et al.
Pubblicazione: (2024)

Rethinking Personalization in Large Language Models at the Token Level
di: Zhang, Chenheng, et al.
Pubblicazione: (2026)

Tokenization Matters! Degrading Large Language Models through Challenging Their Tokenization
di: Wang, Dixuan, et al.
Pubblicazione: (2024)

A Peek into Token Bias: Large Language Models Are Not Yet Genuine Reasoners
di: Jiang, Bowen, et al.
Pubblicazione: (2024)

Disentangling Reasoning Tokens and Boilerplate Tokens For Language Model Fine-tuning
di: Ye, Ziang, et al.
Pubblicazione: (2024)

Understanding and Mitigating Tokenization Bias in Language Models
di: Phan, Buu, et al.
Pubblicazione: (2024)

The Token Tax: Systematic Bias in Multilingual Tokenization
di: Lundin, Jessica M., et al.
Pubblicazione: (2025)

Word Recovery in Large Language Models Enables Character-Level Tokenization Robustness
di: Yang, Zhipeng, et al.
Pubblicazione: (2026)

Remask, Don't Replace: Token-to-Mask Refinement in Diffusion Large Language Models
di: Yao, Lin
Pubblicazione: (2026)

Retrofitting Large Language Models with Dynamic Tokenization
di: Feher, Darius, et al.
Pubblicazione: (2024)

FTP: A Fine-grained Token-wise Pruner for Large Language Models via Token Routing
di: Li, Zekai, et al.
Pubblicazione: (2024)

LFTR: Learning-Free Token Reduction for Multimodal Large Language Models
di: Zhao, Zihui, et al.
Pubblicazione: (2025)

ResT: Reshaping Token-Level Policy Gradients for Tool-Use Large Language Models
di: Lin, Zihan, et al.
Pubblicazione: (2025)

Beyond Token-Level Policy Gradients for Complex Reasoning with Large Language Models
di: Xu, Mufan, et al.
Pubblicazione: (2026)

ReTok: Replacing Tokenizer to Enhance Representation Efficiency in Large Language Model
di: Gu, Shuhao, et al.
Pubblicazione: (2024)

SOUP: Token-level Single-sample Mix-policy Reinforcement Learning for Large Language Models
di: Yang, Lei, et al.
Pubblicazione: (2026)

Identifying and Analyzing Performance-Critical Tokens in Large Language Models
di: Bai, Yu, et al.
Pubblicazione: (2024)

Reasoning Bias of Next Token Prediction Training
di: Lin, Pengxiao, et al.
Pubblicazione: (2025)

Performance Evaluation of Tokenizers in Large Language Models for the Assamese Language
di: Tamang, Sagar, et al.
Pubblicazione: (2024)

LBPE: Long-token-first Tokenization to Improve Large Language Models
di: Lian, Haoran, et al.
Pubblicazione: (2024)

Vision-centric Token Compression in Large Language Model
di: Xing, Ling, et al.
Pubblicazione: (2025)

SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models
di: Zhang, Xin, et al.
Pubblicazione: (2023)

Technical Report: Impact of Position Bias on Language Models in Token Classification
di: Amor, Mehdi Ben, et al.
Pubblicazione: (2023)

Targeted Remasking: Replacing Token Editing with Token-to-Mask Refinement in Discrete Diffusion Language Models
di: Yao, Lin
Pubblicazione: (2026)

Memory Retrieval and Consolidation in Large Language Models through Function Tokens
di: Zhang, Shaohua, et al.
Pubblicazione: (2025)

Token-Level Privacy in Large Language Models
di: Harel, Re'em, et al.
Pubblicazione: (2025)

Token-wise Influential Training Data Retrieval for Large Language Models
di: Lin, Huawei, et al.
Pubblicazione: (2024)

Saliency-driven Dynamic Token Pruning for Large Language Models
di: Tao, Yao, et al.
Pubblicazione: (2025)

Token-Efficient Leverage Learning in Large Language Models
di: Zeng, Yuanhao, et al.
Pubblicazione: (2024)

One Token Can Help! Learning Scalable and Pluggable Virtual Tokens for Retrieval-Augmented Large Language Models
di: Zhu, Yutao, et al.
Pubblicazione: (2024)

Glitch Tokens in Large Language Models: Categorization Taxonomy and Effective Detection
di: Li, Yuxi, et al.
Pubblicazione: (2024)

Tokenization Falling Short: On Subword Robustness in Large Language Models
di: Chai, Yekun, et al.
Pubblicazione: (2024)

Contextually Structured Token Dependency Encoding for Large Language Models
di: Blades, James, et al.
Pubblicazione: (2025)

MorphPiece : A Linguistic Tokenizer for Large Language Models
di: Jabbar, Haris
Pubblicazione: (2023)

Token Signature: Predicting Chain-of-Thought Gains with Token Decoding Feature in Large Language Models
di: Liu, Peijie, et al.
Pubblicazione: (2025)

From Token to Token Pair: Efficient Prompt Compression for Large Language Models in Clinical Prediction
di: Zhu, Mingcheng, et al.
Pubblicazione: (2026)

TASE: Token Awareness and Structured Evaluation for Multilingual Language Models
di: Zhao, Chenzhuo, et al.
Pubblicazione: (2025)