Image de couverture de livre

Reasoning Bias of Next Token Prediction Training

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Lin, Pengxiao, Zhang, Zhongwang, Xu, Zhi-Qin John
Format:	Preprint
Publié:	2025
Sujets:	Computation and Language Machine Learning
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Complexity Control Facilitates Reasoning-Based Compositional Generalization in Transformers
par: Zhang, Zhongwang, et autres
Publié: (2025)

An Analysis for Reasoning Bias of Language Models with Small Initialization
par: Yao, Junjie, et autres
Publié: (2025)

Initialization is Critical to Whether Transformers Fit Composite Functions by Reasoning or Memorizing
par: Zhang, Zhongwang, et autres
Publié: (2024)

Implicit Optimization Bias of Next-Token Prediction in Linear Models
par: Thrampoulidis, Christos
Publié: (2024)

Cautious Next Token Prediction
par: Wang, Yizhou, et autres
Publié: (2025)

Anchor function: a type of benchmark functions for studying language models
par: Zhang, Zhongwang, et autres
Publié: (2024)

ENTP: Encoder-only Next Token Prediction
par: Ewer, Ethan, et autres
Publié: (2024)

On the Bias of Next-Token Predictors Toward Systematically Inefficient Reasoning: A Shortest-Path Case Study
par: Alberghi, Riccardo, et autres
Publié: (2025)

Understanding the Language Model to Solve the Symbolic Multi-Step Reasoning Problem from the Perspective of Buffer Mechanism
par: Wang, Zhiwei, et autres
Publié: (2024)

Beyond Next Token Prediction: Patch-Level Training for Large Language Models
par: Shao, Chenze, et autres
Publié: (2024)

Identity Bridge: Enabling Implicit Reasoning via Shared Latent Memory
par: Lin, Pengxiao, et autres
Publié: (2025)

Efficient Training of Language Models with Compact and Consistent Next Token Distributions
par: Sathe, Ashutosh, et autres
Publié: (2024)

Loss Spike in Training Neural Networks
par: Li, Xiaolong, et autres
Publié: (2023)

Is Next Token Prediction Sufficient for GPT? Exploration on Code Logic Comprehension
par: Qi, Mengnan, et autres
Publié: (2024)

Improving Next Tokens via Second-to-Last Predictions with Generate and Refine
par: Schneider, Johannes
Publié: (2024)

I Predict Therefore I Am: Is Next Token Prediction Enough to Learn Human-Interpretable Concepts from Data?
par: Liu, Yuhang, et autres
Publié: (2025)

Towards Auto-Regressive Next-Token Prediction: In-Context Learning Emerges from Generalization
par: Gong, Zixuan, et autres
Publié: (2025)

Lossless Compression of Large Language Model-Generated Text via Next-Token Prediction
par: Mao, Yu, et autres
Publié: (2025)

On Next-Token Prediction in LLMs: How End Goals Determine the Consistency of Decoding Algorithms
par: Trauger, Jacob, et autres
Publié: (2025)

A Law of Next-Token Prediction in Large Language Models
par: He, Hangfeng, et autres
Publié: (2024)

Differentially Private Next-Token Prediction of Large Language Models
par: Flemings, James, et autres
Publié: (2024)

SENTRA: Selected-Next-Token Transformer for LLM Text Detection
par: Plyler, Mitchell, et autres
Publié: (2025)

Mechanics of Next Token Prediction with Self-Attention
par: Li, Yingcong, et autres
Publié: (2024)

Dynamics of Spontaneous Topic Changes in Next Token Prediction with Self-Attention
par: Jia, Mumin, et autres
Publié: (2025)

Auto-Regressive Next-Token Predictors are Universal Learners
par: Malach, Eran
Publié: (2023)

Beyond Next-Token Prediction: A Performance Characterization of Diffusion versus Autoregressive Language Models
par: Kim, Minseo, et autres
Publié: (2025)

When Is Next-Token Prediction Useful? Marginalization, Ergodicity, Mixture Identifiability, Local Sufficiency, RAG, Tools, and Programming
par: Corielli, Francesco
Publié: (2026)

Probing Geometry of Next Token Prediction Using Cumulant Expansion of the Softmax Entropy
par: Viswanathan, Karthik, et autres
Publié: (2025)

Understanding the Emergence of Seemingly Useless Features in Next-Token Predictors
par: Rofin, Mark, et autres
Publié: (2026)

Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capability
par: Lin, Zicheng, et autres
Publié: (2024)

Multi-Token Prediction via Self-Distillation
par: Kirchenbauer, John, et autres
Publié: (2026)

IAPO: Information-Aware Policy Optimization for Token-Efficient Reasoning
par: He, Yinhan, et autres
Publié: (2026)

Not All Tokens Matter: Towards Efficient LLM Reasoning via Token Significance in Reinforcement Learning
par: Liu, Hanbing, et autres
Publié: (2025)

Training Large Language Models To Reason In Parallel With Global Forking Tokens
par: Jia, Sheng, et autres
Publié: (2025)

Token Homogenization under Positional Bias
par: Yusupov, Viacheslav, et autres
Publié: (2025)

Training-Trajectory-Aware Token Selection
par: Shen, Zhanming, et autres
Publié: (2026)

APOLLO: An Optimized Training Approach for Long-form Numerical Reasoning
par: Sun, Jiashuo, et autres
Publié: (2022)

A Causal World Model Underlying Next Token Prediction: Exploring GPT in a Controlled Environment
par: Rohekar, Raanan Y., et autres
Publié: (2024)

Low-Bit Quantization Favors Undertrained LLMs: Scaling Laws for Quantized LLMs with 100T Training Tokens
par: Ouyang, Xu, et autres
Publié: (2024)

Mitigating Catastrophic Forgetting in Mathematical Reasoning Finetuning through Mixed Training
par: Reynolds, John Graham
Publié: (2025)