Image de couverture de livre

Tokenization Matters! Degrading Large Language Models through Challenging Their Tokenization

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Wang, Dixuan, Li, Yanda, Jiang, Junyuan, Ding, Zepeng, Luo, Ziqin, Jiang, Guochao, Liang, Jiaqing, Yang, Deqing
Format:	Preprint
Publié:	2024
Sujets:	Computation and Language Artificial Intelligence
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

RLAP: A Reinforcement Learning Enhanced Adaptive Planning Framework for Multi-step NLP Task Solving
par: Ding, Zepeng, et autres
Publié: (2025)

Reason from Fallacy: Enhancing Large Language Models' Logical Reasoning through Logical Fallacy Understanding
par: Li, Yanda, et autres
Publié: (2024)

ToNER: Type-oriented Named Entity Recognition with Generative Language Model
par: Jiang, Guochao, et autres
Publié: (2024)

Adaptive Reinforcement Learning Planning: Harnessing Large Language Models for Complex Information Extraction
par: Ding, Zepeng, et autres
Publié: (2024)

Mitigating Out-of-Entity Errors in Named Entity Recognition: A Sentence-Level Strategy
par: Jiang, Guochao, et autres
Publié: (2024)

FlashThink: An Early Exit Method For Efficient Reasoning
par: Jiang, Guochao, et autres
Publié: (2025)

SED: Self-Evaluation Decoding Enhances Large Language Models for Better Generation
par: Luo, Ziqin, et autres
Publié: (2024)

P-ICL: Point In-Context Learning for Named Entity Recognition with Large Language Models
par: Jiang, Guochao, et autres
Publié: (2024)

Past Meets Present: Creating Historical Analogy with Large Language Models
par: Li, Nianqi, et autres
Publié: (2024)

Memory Retrieval and Consolidation in Large Language Models through Function Tokens
par: Zhang, Shaohua, et autres
Publié: (2025)

QUILL: Quotation Generation Enhancement of Large Language Models
par: Xiao, Jin, et autres
Publié: (2024)

Improving Recall of Large Language Models: A Model Collaboration Approach for Relational Triple Extraction
par: Ding, Zepeng, et autres
Publié: (2024)

Enhancing Large Language Models for Mobility Analytics with Semantic Location Tokenization
par: Chen, Yile, et autres
Publié: (2025)

Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capability
par: Lin, Zicheng, et autres
Publié: (2024)

ANALOGYKB: Unlocking Analogical Reasoning of Language Models with A Million-scale Knowledge Base
par: Yuan, Siyu, et autres
Publié: (2023)

Token Signature: Predicting Chain-of-Thought Gains with Token Decoding Feature in Large Language Models
par: Liu, Peijie, et autres
Publié: (2025)

Enhancing Quantitative Reasoning Skills of Large Language Models through Dimension Perception
par: Huang, Yuncheng, et autres
Publié: (2023)

A Peek into Token Bias: Large Language Models Are Not Yet Genuine Reasoners
par: Jiang, Bowen, et autres
Publié: (2024)

Every Token Counts: Generalizing 16M Ultra-Long Context in Large Language Models
par: Hu, Xiang, et autres
Publié: (2025)

Tokenization Matters: Navigating Data-Scarce Tokenization for Gender Inclusive Language Technologies
par: Ovalle, Anaelia, et autres
Publié: (2023)

TEAL: Tokenize and Embed ALL for Multi-modal Large Language Models
par: Yang, Zhen, et autres
Publié: (2023)

Counting Ability of Large Language Models and Impact of Tokenization
par: Zhang, Xiang, et autres
Publié: (2024)

Backdoor Token Unlearning: Exposing and Defending Backdoors in Pretrained Language Models
par: Jiang, Peihai, et autres
Publié: (2025)

One Token Is Enough: Improving Diffusion Language Models with a Sink Token
par: Zhang, Zihou, et autres
Publié: (2026)

Compressing Sequences in the Latent Embedding Space: $K$-Token Merging for Large Language Models
par: Xu, Zihao, et autres
Publié: (2026)

The Path Matters: Learning a Token-Commitment Policy for Diffusion Language Models
par: Sun, Bohang, et autres
Publié: (2026)

Probabilistic Token Alignment for Large Language Model Fusion
par: Zeng, Runjia, et autres
Publié: (2025)

Skeletons Matter: Dynamic Data Augmentation for Text-to-Query
par: Ji, Yuchen, et autres
Publié: (2025)

Re-Initialization Token Learning for Tool-Augmented Large Language Models
par: Li, Chenghao, et autres
Publié: (2025)

Tokenization Matters: Improving Zero-Shot NER for Indic Languages
par: Pattnayak, Priyaranjan, et autres
Publié: (2025)

Thinking Tokens for Language Modeling
par: Herel, David, et autres
Publié: (2024)

SpeechR: A Benchmark for Speech Reasoning in Large Audio-Language Models
par: Yang, Wanqi, et autres
Publié: (2025)

Tokenization, Fusion and Decoupling: Bridging the Granularity Mismatch Between Large Language Models and Knowledge Graphs
par: Su, Siyue, et autres
Publié: (2026)

Contextual Reinforcement in Multimodal Token Compression for Large Language Models
par: Piero, Naderdel, et autres
Publié: (2025)

Saliency-driven Dynamic Token Pruning for Large Language Models
par: Tao, Yao, et autres
Publié: (2025)

Is There a Case for Conversation Optimized Tokenizers in Large Language Models?
par: Ferrando, Raquel, et autres
Publié: (2025)

Counterfactual Token Generation in Large Language Models
par: Chatzi, Ivi, et autres
Publié: (2024)

Cornerstones or Stumbling Blocks? Deciphering the Rock Tokens in On-Policy Distillation
par: Jiang, Yuxuan, et autres
Publié: (2026)

SemToken: Semantic-Aware Tokenization for Efficient Long-Context Language Modeling
par: Liu, Dong, et autres
Publié: (2025)

Token-Supervised Value Models for Enhancing Mathematical Problem-Solving Capabilities of Large Language Models
par: Lee, Jung Hyun, et autres
Publié: (2024)