Cover Image

FLEXITOKENS: Flexible Tokenization for Evolving Language Models

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Owodunni, Abraham Toluwase, Ahia, Orevaoghene, Kumar, Sachin
Format:	Preprint
Published:	2025
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Continually Adding New Languages to Multilingual Language Models
by: Owodunni, Abraham Toluwase, et al.
Published: (2025)

MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenization
by: Ahia, Orevaoghene, et al.
Published: (2024)

Broken Tokens? Your Language Model can Secretly Handle Non-Canonical Tokenizations
by: Zheng, Brian Siyuan, et al.
Published: (2025)

BASS: Benchmarking Audio LMs for Musical Structure and Semantic Reasoning
by: Jang, Min, et al.
Published: (2026)

MYTE: Morphology-Driven Byte Encoding for Better and Fairer Multilingual Language Modeling
by: Limisiewicz, Tomasz, et al.
Published: (2024)

Extracting Lexical Features from Dialects via Interpretable Dialect Classifiers
by: Xie, Roy, et al.
Published: (2024)

AccentFold: A Journey through African Accents for Zero-Shot ASR Adaptation to Target Accents
by: Owodunni, Abraham Toluwase, et al.
Published: (2024)

DIALECTBENCH: A NLP Benchmark for Dialects, Varieties, and Closely-Related Languages
by: Faisal, Fahim, et al.
Published: (2024)

Critical Learning Periods: Leveraging Early Training Dynamics for Efficient Data Pruning
by: Chimoto, Everlyn Asiko, et al.
Published: (2024)

Teaching LLMs to Abstain across Languages via Multilingual Feedback
by: Feng, Shangbin, et al.
Published: (2024)

Voices Unheard: NLP Resources and Models for Yorùbá Regional Dialects
by: Ahia, Orevaoghene, et al.
Published: (2024)

IDIOLEX: Unified and Continuous Representations for Idiolectal and Stylistic Variation
by: Kantharuban, Anjali, et al.
Published: (2026)

UbuntuGuard: A Culturally-Grounded Policy Benchmark for Equitable AI Safety in African Languages
by: Abdullahi, Tassallah, et al.
Published: (2026)

Meta-Tool: Efficient Few-Shot Tool Adaptation for Small Language Models
by: Kumar, Sachin
Published: (2026)

Performant ASR Models for Medical Entities in Accented Speech
by: Afonja, Tejumade, et al.
Published: (2024)

Flexibly Scaling Large Language Models Contexts Through Extensible Tokenization
by: Shao, Ninglu, et al.
Published: (2024)

Think before you speak: Training Language Models With Pause Tokens
by: Goyal, Sachin, et al.
Published: (2023)

GroundCocoa: A Benchmark for Evaluating Compositional & Conditional Reasoning in Language Models
by: Kohli, Harsh, et al.
Published: (2024)

Overriding Safety protections of Open-source Models
by: Kumar, Sachin
Published: (2024)

A Comparative Empirical Study of Catastrophic Forgetting Mitigation in Sequential Task Adaptation for Continual Natural Language Processing Systems
by: Abrahamyan, Aram, et al.
Published: (2026)

Efficient Vision-Language Models by Summarizing Visual Tokens into Compact Registers
by: Wen, Yuxin, et al.
Published: (2024)

TESS 2: A Large-Scale Generalist Diffusion Language Model
by: Tae, Jaesung, et al.
Published: (2025)

Fine-Grained Emotion Detection on GoEmotions: Experimental Comparison of Classical Machine Learning, BiLSTM, and Transformer Models
by: Harutyunyan, Ani, et al.
Published: (2026)

Achieving Tokenizer Flexibility in Language Models through Heuristic Adaptation and Supertoken Learning
by: Sharthak, Shaurya, et al.
Published: (2025)

BLAB: Brutally Long Audio Bench
by: Ahia, Orevaoghene, et al.
Published: (2025)

Problematic Tokens: Tokenizer Bias in Large Language Models
by: Yang, Jin, et al.
Published: (2024)

ComPO: Community Preferences for Language Model Personalization
by: Kumar, Sachin, et al.
Published: (2024)

The NaijaVoices Dataset: Cultivating Large-Scale, High-Quality, Culturally-Rich Speech Data for African Languages
by: Emezue, Chris, et al.
Published: (2025)

Steering off Course: Reliability Challenges in Steering Language Models
by: Da Silva, Patrick Queiroz, et al.
Published: (2025)

Rethinking Tokenization: Crafting Better Tokenizers for Large Language Models
by: Yang, Jinbiao
Published: (2024)

On the Generalization Gap in Self-Evolving Language Model Reasoning
by: Qi, Zhenting, et al.
Published: (2026)

Compute Optimal Tokenization
by: Limisiewicz, Tomasz, et al.
Published: (2026)

Disentangling Reasoning Tokens and Boilerplate Tokens For Language Model Fine-tuning
by: Ye, Ziang, et al.
Published: (2024)

Broken Words, Broken Performance: Effect of Tokenization on Performance of LLMs
by: Pawar, Sachin, et al.
Published: (2025)

Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research
by: Soldaini, Luca, et al.
Published: (2024)

Reasoning Up the Instruction Ladder for Controllable Language Models
by: Zheng, Zishuo, et al.
Published: (2025)

Rethinking Experience Utilization in Self-Evolving Language Model Agents
by: Zhao, Weixiang, et al.
Published: (2026)

Evolving Knowledge Distillation with Large Language Models and Active Learning
by: Liu, Chengyuan, et al.
Published: (2024)

Token Weighting for Long-Range Language Modeling
by: Helm, Falko, et al.
Published: (2025)

Large Language Model as Token Compressor and Decompressor
by: Li, Wenbing, et al.
Published: (2026)