Cover Image

Achieving Tokenizer Flexibility in Language Models through Heuristic Adaptation and Supertoken Learning

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Sharthak, Shaurya, Pahalwan, Vinayak, Kamath, Adithya, Shirawalmath, Adarsh
Format:	Preprint
Published:	2025
Subjects:	Computation and Language Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

From Bytes to Borsch: Fine-Tuning Gemma and Mistral for the Ukrainian Language Representation
by: Kiulian, Artur, et al.
Published: (2024)

Flexora: Flexible Low Rank Adaptation for Large Language Models
by: Wei, Chenxing, et al.
Published: (2024)

Tokenization Matters! Degrading Large Language Models through Challenging Their Tokenization
by: Wang, Dixuan, et al.
Published: (2024)

Examining Alignment of Large Language Models through Representative Heuristics: The Case of Political Stereotypes
by: Jeoung, Sullam, et al.
Published: (2025)

Scope Ambiguities in Large Language Models
by: Kamath, Gaurav, et al.
Published: (2024)

Language Models Largely Exhibit Human-like Constituent Ordering Preferences
by: Tur, Ada Defne, et al.
Published: (2025)

Memory Retrieval and Consolidation in Large Language Models through Function Tokens
by: Zhang, Shaohua, et al.
Published: (2025)

Thinking Tokens for Language Modeling
by: Herel, David, et al.
Published: (2024)

NADIR: Differential Attention Flow for Non-Autoregressive Transliteration in Indic Languages
by: Tomar, Lakshya, et al.
Published: (2026)

Achieving Sparse Activation in Small Language Models
by: Song, Jifeng, et al.
Published: (2024)

Large Language Models as Zero-shot Dialogue State Tracker through Function Calling
by: Li, Zekun, et al.
Published: (2024)

Re-Initialization Token Learning for Tool-Augmented Large Language Models
by: Li, Chenghao, et al.
Published: (2025)

Flexible Realignment of Language Models
by: Zhu, Wenhong, et al.
Published: (2025)

Learning Faster with Better Tokens: Parameter-Efficient Vocabulary Adaptation for Specialized Text Summarization
by: Balde, Gunjan, et al.
Published: (2026)

Accelerating Multilingual Language Model for Excessively Tokenized Languages
by: Hong, Jimin, et al.
Published: (2024)

From Next-Token to Mathematics: The Learning Dynamics of Mathematical Reasoning in Language Models
by: Mishra, Shubhra, et al.
Published: (2024)

The Path Matters: Learning a Token-Commitment Policy for Diffusion Language Models
by: Sun, Bohang, et al.
Published: (2026)

SemToken: Semantic-Aware Tokenization for Efficient Long-Context Language Modeling
by: Liu, Dong, et al.
Published: (2025)

One Token Is Enough: Improving Diffusion Language Models with a Sink Token
by: Zhang, Zihou, et al.
Published: (2026)

From Language Models over Tokens to Language Models over Characters
by: Vieira, Tim, et al.
Published: (2024)

SGSH: Stimulate Large Language Models with Skeleton Heuristics for Knowledge Base Question Generation
by: Guo, Shasha, et al.
Published: (2024)

DELIA: Diversity-Enhanced Learning for Instruction Adaptation in Large Language Models
by: Zeng, Yuanhao, et al.
Published: (2024)

Domain-Adaptation through Synthetic Data: Fine-Tuning Large Language Models for German Law
by: Bashir, Ali Hamza, et al.
Published: (2026)

PathWise: Planning through World Model for Automated Heuristic Design via Self-Evolving LLMs
by: Gungordu, Oguzhan, et al.
Published: (2026)

AlphaToken: Decoupling Adaptation and Stability for Path-Aware Response Token Valuation in LLM Post-Training
by: Qing, Liu, et al.
Published: (2026)

Regional Tiny Stories: Using Small Models to Compare Language Learning and Tokenizer Performance
by: Patil, Nirvan, et al.
Published: (2025)

Small Models, Big Results: Achieving Superior Intent Extraction through Decomposition
by: Cohen, Danielle, et al.
Published: (2025)

DaSAThco: Data-Aware SAT Heuristics Combinations Optimization via Large Language Models
by: Chen, Minyu, et al.
Published: (2025)

No LLM is Free From Bias: A Comprehensive Study of Bias Evaluation in Large Language Models
by: Kumar, Charaka Vinayak, et al.
Published: (2025)

Thunder-Tok: Minimizing Tokens per Word in Tokenizing Korean Texts for Generative Language Models
by: Cho, Gyeongje, et al.
Published: (2025)

Token Signature: Predicting Chain-of-Thought Gains with Token Decoding Feature in Large Language Models
by: Liu, Peijie, et al.
Published: (2025)

Targeted Remasking: Replacing Token Editing with Token-to-Mask Refinement in Discrete Diffusion Language Models
by: Yao, Lin
Published: (2026)

Token-Efficient Leverage Learning in Large Language Models
by: Zeng, Yuanhao, et al.
Published: (2024)

Counting Ability of Large Language Models and Impact of Tokenization
by: Zhang, Xiang, et al.
Published: (2024)

The Power of Adaptation: Boosting In-Context Learning through Adaptive Prompting
by: Cai, Shuzhang, et al.
Published: (2024)

The Tokenization Bottleneck: How Vocabulary Extension Improves Chemistry Representation Learning in Pretrained Language Models
by: Kalamkar, Prathamesh, et al.
Published: (2025)

Achieving Peak Performance for Large Language Models: A Systematic Review
by: Rostam, Zhyar Rzgar K, et al.
Published: (2024)

TildeOpen LLM: Leveraging Curriculum Learning to Achieve Equitable Language Representation
by: Bergmanis, Toms, et al.
Published: (2026)

LLMs Learn Task Heuristics from Demonstrations: A Heuristic-Driven Prompting Strategy for Document-Level Event Argument Extraction
by: Zhou, Hanzhang, et al.
Published: (2023)

Evaluating Tokenizer Performance of Large Language Models Across Official Indian Languages
by: Tamang, S., et al.
Published: (2024)