Image de couverture de livre

Training LLMs Beyond Next Token Prediction -- Filling the Mutual Information Gap

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Yang, Chun-Hao, Feng, Bo-Han, Lai, Tzu-Yuan, Chen, Yan Yu, Huang, Yin-Kai Dean, Lin, Shou-De
Format:	Preprint
Publié:	2025
Sujets:	Computation and Language Artificial Intelligence
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

_version_	1866911247337783296
author	Yang, Chun-Hao Feng, Bo-Han Lai, Tzu-Yuan Chen, Yan Yu Huang, Yin-Kai Dean Lin, Shou-De
author_facet	Yang, Chun-Hao Feng, Bo-Han Lai, Tzu-Yuan Chen, Yan Yu Huang, Yin-Kai Dean Lin, Shou-De
contents	Optimizing training performance in large language models (LLMs) remains an essential challenge, particularly in improving model performance while maintaining computational costs. This work challenges the conventional approach of training LLMs using next-token prediction (NTP), arguing that by predicting information-rich tokens during training, there is a more effective way to train LLMs. We investigate the impact of the proposed solution in three kinds of tasks for LLMs: arithmetic, multi-label classification of text, and natural-language generation. This work offers a principled approach to optimizing LLM training, advancing both model performance and theoretical understanding of the target-token selection strategies.
format	Preprint
id	arxiv_https___arxiv_org_abs_2511_00198
institution	arXiv
publishDate	2025
record_format	arxiv
spellingShingle	Training LLMs Beyond Next Token Prediction -- Filling the Mutual Information Gap Yang, Chun-Hao Feng, Bo-Han Lai, Tzu-Yuan Chen, Yan Yu Huang, Yin-Kai Dean Lin, Shou-De Computation and Language Artificial Intelligence Optimizing training performance in large language models (LLMs) remains an essential challenge, particularly in improving model performance while maintaining computational costs. This work challenges the conventional approach of training LLMs using next-token prediction (NTP), arguing that by predicting information-rich tokens during training, there is a more effective way to train LLMs. We investigate the impact of the proposed solution in three kinds of tasks for LLMs: arithmetic, multi-label classification of text, and natural-language generation. This work offers a principled approach to optimizing LLM training, advancing both model performance and theoretical understanding of the target-token selection strategies.
title	Training LLMs Beyond Next Token Prediction -- Filling the Mutual Information Gap
topic	Computation and Language Artificial Intelligence
url	https://arxiv.org/abs/2511.00198