Cover Image

Training LLMs Beyond Next Token Prediction -- Filling the Mutual Information Gap

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Yang, Chun-Hao, Feng, Bo-Han, Lai, Tzu-Yuan, Chen, Yan Yu, Huang, Yin-Kai Dean, Lin, Shou-De
Format:	Preprint
Published:	2025
Subjects:	Computation and Language Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Reasoning Bias of Next Token Prediction Training
by: Lin, Pengxiao, et al.
Published: (2025)

lil'HDoC: An Algorithm for Good Arm Identification under Small Threshold Gap
by: Tsai, Tzu-Hsien, et al.
Published: (2024)

Beyond Next Token Prediction: Patch-Level Training for Large Language Models
by: Shao, Chenze, et al.
Published: (2024)

Narrative-Centered Emotional Reflection: Scaffolding Autonomous Emotional Literacy with AI
by: Han, Shou-Tzu
Published: (2025)

Beyond Next-Token: Next-X Prediction for Autoregressive Visual Generation
by: Ren, Sucheng, et al.
Published: (2025)

NEST-RQ: Next Token Prediction for Speech Self-Supervised Pre-Training
by: Han, Minglun, et al.
Published: (2024)

Object Recognition as Next Token Prediction
by: Yue, Kaiyu, et al.
Published: (2023)

Beyond Next-Token Alignment: Distilling Multimodal Large Language Models via Token Interactions
by: Chen, Lin, et al.
Published: (2026)

Investigation of the Properties of Gap‐Filling Materials Enhanced With Nanofillers
by: Hao Huang, et al.
Published: (2025)

Grounding-Aware Token Pruning: Recovering from Drastic Performance Drops in Visual Grounding Caused by Pruning
by: Chien, Tzu-Chun, et al.
Published: (2025)

Cautious Next Token Prediction
by: Wang, Yizhou, et al.
Published: (2025)

Beyond the Next Token: Towards Prompt-Robust Zero-Shot Classification via Efficient Multi-Token Prediction
by: Qian, Junlang, et al.
Published: (2025)

From Next-Token to Next-Block: A Principled Adaptation Path for Diffusion LLMs
by: Tian, Yuchuan, et al.
Published: (2025)

Beyond Tokens: Concept-Level Training Objectives for LLMs
by: Iyer, Laya, et al.
Published: (2026)

Knowledge Card: Filling LLMs' Knowledge Gaps with Plug-in Specialized Language Models
by: Feng, Shangbin, et al.
Published: (2023)

Bridging the Training-Inference Gap in LLMs by Leveraging Self-Generated Tokens
by: Cen, Zhepeng, et al.
Published: (2024)

Prot2Token: A Unified Framework for Protein Modeling via Next-Token Prediction
by: Pourmirzaei, Mahdi, et al.
Published: (2025)

LLMs are Not Just Next Token Predictors
by: Downes, Stephen M., et al.
Published: (2024)

Moving Beyond Next-Token Prediction: Transformers are Context-Sensitive Language Generators
by: Rhee, Phill Kyu
Published: (2025)

PanoLlama: Generating Endless and Coherent Panoramas with Next-Token-Prediction LLMs
by: Zhou, Teng, et al.
Published: (2024)

Next-Token Prediction Task Assumes Optimal Data Ordering for LLM Training in Proof Generation
by: An, Chenyang, et al.
Published: (2024)

Lossless Compression of Large Language Model-Generated Text via Next-Token Prediction
by: Mao, Yu, et al.
Published: (2025)

Deep Learning-Based Prediction of Suspension Dynamics Performance in Multi-Axle Vehicles
by: Lin, Kai Chun, et al.
Published: (2024)

Beyond Multi-Token Prediction: Pretraining LLMs with Future Summaries
by: Mahajan, Divyat, et al.
Published: (2025)

Union-Intersection Union-Find for Decoding Depolarizing Errors in Topological Codes
by: Lin, Tzu-Hao, et al.
Published: (2025)

Analysis of Centrifugal Clutches in Two-Speed Automatic Transmissions with Deep Learning-Based Engagement Prediction
by: Lin, Bo-Yi, et al.
Published: (2024)

Next-Token Prediction and Regret Minimization
by: Mohri, Mehryar, et al.
Published: (2026)

Humanoid Locomotion as Next Token Prediction
by: Radosavovic, Ilija, et al.
Published: (2024)

SGG-R$^{\rm 3}$: From Next-Token Prediction to End-to-End Unbiased Scene Graph Generation
by: Feng, Jiaye, et al.
Published: (2026)

On Next-Token Prediction in LLMs: How End Goals Determine the Consistency of Decoding Algorithms
by: Trauger, Jacob, et al.
Published: (2025)

A Case Study of Next Portfolio Prediction for Mutual Funds
by: Thomaz, Guilherme, et al.
Published: (2024)

Differential Good Arm Identification
by: Tsai, Yun-Da, et al.
Published: (2023)

BrainVista: Modeling Naturalistic Brain Dynamics as Multimodal Next-Token Prediction
by: Yin, Xuanhua, et al.
Published: (2026)

NITP: Next Implicit Token Prediction for LLM Pre-training
by: Zhang, Xiangdong, et al.
Published: (2026)

Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction
by: Yang, Shu-wen, et al.
Published: (2025)

MIGA: Mutual Information-Guided Attack on Denoising Models for Semantic Manipulation
by: Li, Guanghao, et al.
Published: (2025)

Beyond Next-Token Prediction: A Performance Characterization of Diffusion versus Autoregressive Language Models
by: Kim, Minseo, et al.
Published: (2025)

Doc2Token: Bridging Vocabulary Gap by Predicting Missing Tokens for E-commerce Search
by: Li, Kaihao, et al.
Published: (2024)

AMP: Autoregressive Motion Prediction Revisited with Next Token Prediction for Autonomous Driving
by: Jia, Xiaosong, et al.
Published: (2024)

Voice-based AI Agents: Filling the Economic Gaps in Digital Health Delivery
by: Wen, Bo, et al.
Published: (2025)