Imagen de Portada

Length-MAX Tokenizer for Language Models

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Dong, Dong, Su, Weijie
Formato:	Preprint
Publicado:	2025
Materias:	Computation and Language Artificial Intelligence Machine Learning
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

A Law of Next-Token Prediction in Large Language Models
por: He, Hangfeng, et al.
Publicado: (2024)

DLER: Doing Length pEnalty Right - Incentivizing More Intelligence per Token via Reinforcement Learning
por: Liu, Shih-Yang, et al.
Publicado: (2025)

Multiplex Thinking: Reasoning via Token-wise Branch-and-Merge
por: Tang, Yao, et al.
Publicado: (2026)

On the Optimal Reasoning Length for RL-Trained Language Models
por: Nohara, Daisuke, et al.
Publicado: (2026)

TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection
por: Wu, Wei, et al.
Publicado: (2024)

Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning
por: Su, DiJia, et al.
Publicado: (2025)

Confidence Regularized Masked Language Modeling using Text Length
por: Ji, Seunghyun, et al.
Publicado: (2025)

Understanding and Mitigating Tokenization Bias in Language Models
por: Phan, Buu, et al.
Publicado: (2024)

Guiding Language Model Reasoning with Planning Tokens
por: Wang, Xinyi, et al.
Publicado: (2023)

Adapting Language Models via Token Translation
por: Feng, Zhili, et al.
Publicado: (2024)

Counterfactual Token Generation in Large Language Models
por: Chatzi, Ivi, et al.
Publicado: (2024)

Adaptive Layer Selection for Layer-Wise Token Pruning in LLM Inference
por: Taniguchi, Rei, et al.
Publicado: (2026)

On the Expressiveness and Length Generalization of Selective State-Space Models on Regular Languages
por: Terzić, Aleksandar, et al.
Publicado: (2024)

Reward Collapse in Aligning Large Language Models
por: Song, Ziang, et al.
Publicado: (2023)

Revisiting Graph-Tokenizing Large Language Models: A Systematic Evaluation of Graph Token Understanding
por: Zhang, Zhongjian, et al.
Publicado: (2026)

Probabilistic Token Alignment for Large Language Model Fusion
por: Zeng, Runjia, et al.
Publicado: (2025)

Specialised or Generic? Tokenization Choices for Radiology Language Models
por: Warr, Hermione, et al.
Publicado: (2025)

Compositional Steering of Large Language Models with Steering Tokens
por: Radevski, Gorjan, et al.
Publicado: (2026)

Token-Efficient Leverage Learning in Large Language Models
por: Zeng, Yuanhao, et al.
Publicado: (2024)

Language Model Cascades: Token-level uncertainty and beyond
por: Gupta, Neha, et al.
Publicado: (2024)

Does Visual Rendering Bypass Tokenization? Investigating Script-Tokenizer Misalignment in Pixel-Based Language Models
por: Susanto, Lucky, et al.
Publicado: (2026)

LaSeR: Reinforcement Learning with Last-Token Self-Rewarding
por: Yang, Wenkai, et al.
Publicado: (2025)

Context-Aware Initialization for Reducing Generative Path Length in Diffusion Language Models
por: Miao, Tongyuan, et al.
Publicado: (2025)

Softplus Attention with Re-weighting Boosts Length Extrapolation in Large Language Models
por: Gao, Bo, et al.
Publicado: (2025)

An Overview of Large Language Models for Statisticians
por: Ji, Wenlong, et al.
Publicado: (2025)

Inconsistent Tokenizations Cause Language Models to be Perplexed by Japanese Grammar
por: Gambardella, Andrew, et al.
Publicado: (2025)

Evaluation of Large Language Models via Coupled Token Generation
por: Benz, Nina Corvelo, et al.
Publicado: (2025)

Scalable Token-Level Hallucination Detection in Large Language Models
por: Min, Rui, et al.
Publicado: (2026)

How Important Is Tokenization in French Medical Masked Language Models?
por: Labrak, Yanis, et al.
Publicado: (2024)

Multiscale Byte Language Models -- A Hierarchical Architecture for Causal Million-Length Sequence Modeling
por: Egli, Eric, et al.
Publicado: (2025)

Cautious Next Token Prediction
por: Wang, Yizhou, et al.
Publicado: (2025)

APPL: A Prompt Programming Language for Harmonious Integration of Programs and Large Language Model Prompts
por: Dong, Honghua, et al.
Publicado: (2024)

TLPO: Token-Level Policy Optimization for Mitigating Language Confusion in Large Language Models
por: Choo, Jinho, et al.
Publicado: (2026)

Training Large Language Models To Reason In Parallel With Global Forking Tokens
por: Jia, Sheng, et al.
Publicado: (2025)

SupraTok: Cross-Boundary Tokenization for Enhanced Language Model Performance
por: Tănase, Andrei-Valentin, et al.
Publicado: (2025)

Unraveling Token Prediction Refinement and Identifying Essential Layers in Language Models
por: Kongmanee, Jaturong
Publicado: (2025)

Semantic Token Clustering for Efficient Uncertainty Quantification in Large Language Models
por: Cao, Qi, et al.
Publicado: (2026)

xVal: A Continuous Numerical Tokenization for Scientific Language Models
por: Golkar, Siavash, et al.
Publicado: (2023)

LangTopo: Aligning Language Descriptions of Graphs with Tokenized Topological Modeling
por: Guan, Zhong, et al.
Publicado: (2024)

Think before you speak: Training Language Models With Pause Tokens
por: Goyal, Sachin, et al.
Publicado: (2023)