Cover Image

When Is Next-Token Prediction Useful? Marginalization, Ergodicity, Mixture Identifiability, Local Sufficiency, RAG, Tools, and Programming

Fuente: arXiv

Saved in:

Bibliographic Details
Main Author:	Corielli, Francesco
Format:	Preprint
Published:	2026
Subjects:	Computation and Language Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

The Need for an External Observer Formalizing the Sufficiency Gap: A Mathematical Extension of Mixture Identifiability and Contextual Grounding in Sequence Models
by: Corielli, Francesco
Published: (2026)

Is Next Token Prediction Sufficient for GPT? Exploration on Code Logic Comprehension
by: Qi, Mengnan, et al.
Published: (2024)

ENTP: Encoder-only Next Token Prediction
by: Ewer, Ethan, et al.
Published: (2024)

Reasoning Bias of Next Token Prediction Training
by: Lin, Pengxiao, et al.
Published: (2025)

Cautious Next Token Prediction
by: Wang, Yizhou, et al.
Published: (2025)

Implicit Optimization Bias of Next-Token Prediction in Linear Models
by: Thrampoulidis, Christos
Published: (2024)

Improving Next Tokens via Second-to-Last Predictions with Generate and Refine
by: Schneider, Johannes
Published: (2024)

Towards Auto-Regressive Next-Token Prediction: In-Context Learning Emerges from Generalization
by: Gong, Zixuan, et al.
Published: (2025)

Lossless Compression of Large Language Model-Generated Text via Next-Token Prediction
by: Mao, Yu, et al.
Published: (2025)

On Next-Token Prediction in LLMs: How End Goals Determine the Consistency of Decoding Algorithms
by: Trauger, Jacob, et al.
Published: (2025)

A Law of Next-Token Prediction in Large Language Models
by: He, Hangfeng, et al.
Published: (2024)

Differentially Private Next-Token Prediction of Large Language Models
by: Flemings, James, et al.
Published: (2024)

I Predict Therefore I Am: Is Next Token Prediction Enough to Learn Human-Interpretable Concepts from Data?
by: Liu, Yuhang, et al.
Published: (2025)

Mechanics of Next Token Prediction with Self-Attention
by: Li, Yingcong, et al.
Published: (2024)

Dynamics of Spontaneous Topic Changes in Next Token Prediction with Self-Attention
by: Jia, Mumin, et al.
Published: (2025)

Auto-Regressive Next-Token Predictors are Universal Learners
by: Malach, Eran
Published: (2023)

Beyond Next Token Prediction: Patch-Level Training for Large Language Models
by: Shao, Chenze, et al.
Published: (2024)

Probing Geometry of Next Token Prediction Using Cumulant Expansion of the Softmax Entropy
by: Viswanathan, Karthik, et al.
Published: (2025)

When is Tree Search Useful for LLM Planning? It Depends on the Discriminator
by: Chen, Ziru, et al.
Published: (2024)

Understanding the Emergence of Seemingly Useless Features in Next-Token Predictors
by: Rofin, Mark, et al.
Published: (2026)

SENTRA: Selected-Next-Token Transformer for LLM Text Detection
by: Plyler, Mitchell, et al.
Published: (2025)

Learning to Reason with Mixture of Tokens
by: Jain, Adit, et al.
Published: (2025)

Unraveling Token Prediction Refinement and Identifying Essential Layers in Language Models
by: Kongmanee, Jaturong
Published: (2025)

Efficient Training of Language Models with Compact and Consistent Next Token Distributions
by: Sathe, Ashutosh, et al.
Published: (2024)

Is In-Context Learning Sufficient for Instruction Following in LLMs?
by: Zhao, Hao, et al.
Published: (2024)

Mixture of Tokens: Continuous MoE through Cross-Example Aggregation
by: Antoniak, Szymon, et al.
Published: (2023)

RoseRAG: Robust Retrieval-augmented Generation with Small-scale LLMs via Margin-aware Preference Optimization
by: Liu, Tianci, et al.
Published: (2025)

Beyond Next-Token Prediction: A Performance Characterization of Diffusion versus Autoregressive Language Models
by: Kim, Minseo, et al.
Published: (2025)

Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation
by: Bae, Sangmin, et al.
Published: (2025)

RAG in the Wild: On the (In)effectiveness of LLMs with Mixture-of-Knowledge Retrieval Augmentation
by: Xu, Ran, et al.
Published: (2025)

A Causal World Model Underlying Next Token Prediction: Exploring GPT in a Controlled Environment
by: Rohekar, Raanan Y., et al.
Published: (2024)

Data Mixture Inference: What do BPE Tokenizers Reveal about their Training Data?
by: Hayase, Jonathan, et al.
Published: (2024)

When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models
by: Yoon, Youngsik, et al.
Published: (2026)

Optimizing Attention with Mirror Descent: Generalized Max-Margin Token Selection
by: Julistiono, Addison Kristanto, et al.
Published: (2024)

Parallel Token Prediction for Language Models
by: Draxler, Felix, et al.
Published: (2025)

Mixture of Weight-shared Heterogeneous Group Attention Experts for Dynamic Token-wise KV Optimization
by: Song, Guanghui, et al.
Published: (2025)

Multimodal Latent Language Modeling with Next-Token Diffusion
by: Sun, Yutao, et al.
Published: (2024)

TokenButler: Token Importance is Predictable
by: Akhauri, Yash, et al.
Published: (2025)

Necessary and Sufficient Watermark for Large Language Models
by: Takezawa, Yuki, et al.
Published: (2023)

Multi-Token Prediction via Self-Distillation
by: Kirchenbauer, John, et al.
Published: (2026)