Image de couverture de livre

Beyond Multi-Token Prediction: Pretraining LLMs with Future Summaries

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Mahajan, Divyat, Goyal, Sachin, Idrissi, Badr Youbi, Pezeshki, Mohammad, Mitliagkas, Ioannis, Lopez-Paz, David, Ahuja, Kartik
Format:	Preprint
Publié:	2025
Sujets:	Machine Learning Artificial Intelligence
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Compositional Risk Minimization
par: Mahajan, Divyat, et autres
Publié: (2024)

Towards efficient representation identification in supervised learning
par: Ahuja, Kartik, et autres
Publié: (2022)

Empirical Analysis of Model Selection for Heterogeneous Causal Effect Estimation
par: Mahajan, Divyat, et autres
Publié: (2022)

Distilled Pretraining: A modern lens of Data, In-Context Learning and Test-Time Scaling
par: Goyal, Sachin, et autres
Publié: (2025)

Iterative Amortized Inference: Unifying In-Context Learning and Learned Optimizers
par: Mittal, Sarthak, et autres
Publié: (2025)

ReasonCACHE: Teaching LLMs To Reason Without Weight Updates
par: Gupta, Sharut, et autres
Publié: (2026)

Better & Faster Large Language Models via Multi-token Prediction
par: Gloeckle, Fabian, et autres
Publié: (2024)

From Bytes to Ideas: Language Modeling with Autoregressive U-Nets
par: Videau, Mathurin, et autres
Publié: (2025)

Learning to Defer for Causal Discovery with Imperfect Experts
par: Clivio, Oscar, et autres
Publié: (2025)

Interventional Causal Representation Learning
par: Ahuja, Kartik, et autres
Publié: (2022)

Temperature Matters: Enhancing Watermark Robustness Against Paraphrasing Attacks
par: Idrissi, Badr Youbi, et autres
Publié: (2025)

Unveiling Simplicities of Attention: Adaptive Long-Context Head Identification
par: Donhauser, Konstantin, et autres
Publié: (2025)

The Pitfalls of Memorization: When Memorization Hurts Generalization
par: Bayat, Reza, et autres
Publié: (2024)

Feature learning as alignment: a structural property of gradient descent in non-linear neural networks
par: Beaglehole, Daniel, et autres
Publié: (2024)

Discovering environments with XRM
par: Pezeshki, Mohammad, et autres
Publié: (2023)

Evaluating Interventional Reasoning Capabilities of Large Language Models
par: Kasetty, Tejas, et autres
Publié: (2024)

An Empirical Study of Pre-trained Model Selection for Out-of-Distribution Generalization and Calibration
par: Naganuma, Hiroki, et autres
Publié: (2023)

KV Prediction for Improved Time to First Token
par: Horton, Maxwell, et autres
Publié: (2024)

Performance Control in Early Exiting to Deploy Large Models at the Same Cost of Smaller Ones
par: Mofakhami, Mehrnaz, et autres
Publié: (2024)

Performative Prediction with Neural Networks
par: Mofakhami, Mehrnaz, et autres
Publié: (2023)

Training LLMs Beyond Next Token Prediction -- Filling the Mutual Information Gap
par: Yang, Chun-Hao, et autres
Publié: (2025)

Beyond Line-Level Filtering for the Pretraining Corpora of LLMs
par: Park, Chanwoo, et autres
Publié: (2025)

Inference Optimal VLMs Need Fewer Visual Tokens and More Parameters
par: Li, Kevin Y., et autres
Publié: (2024)

Expecting The Unexpected: Towards Broad Out-Of-Distribution Detection
par: Guille-Escuret, Charles, et autres
Publié: (2023)

Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs
par: Tavakoli, Mohammad, et autres
Publié: (2025)

MAP's not dead yet: Uncovering true language model modes by conditioning away degeneracy
par: Yoshida, Davis, et autres
Publié: (2023)

Pair-In, Pair-Out: Latent Multi-Token Prediction for Efficient LLMs
par: Tan, Wenhui, et autres
Publié: (2026)

Rethinking Thinking Tokens: LLMs as Improvement Operators
par: Madaan, Lovish, et autres
Publié: (2025)

Think before you speak: Training Language Models With Pause Tokens
par: Goyal, Sachin, et autres
Publié: (2023)

A Derandomization Framework for Structure Discovery: Applications in Neural Networks and Beyond
par: Tsikouras, Nikos, et autres
Publié: (2025)

Navigating Potholes with Geometry-Aware Sharpness Minimization
par: Dufort-Labbé, Simon, et autres
Publié: (2026)

TECP: Token-Entropy Conformal Prediction for LLMs
par: Xu, Beining, et autres
Publié: (2025)

On the Identifiability of Quantized Factors
par: Barin-Pacela, Vitória, et autres
Publié: (2023)

Efficient Joint Prediction of Multiple Future Tokens
par: Ahn, Kwangjun, et autres
Publié: (2025)

Beyond Shapley Values: Cooperative Games for the Interpretation of Machine Learning Models
par: Idrissi, Marouane Il, et autres
Publié: (2025)

Supervised Pretraining for Material Property Prediction
par: Rahman, Chowdhury Mohammad Abid, et autres
Publié: (2025)

From Token to Rhythm: A Multi-Scale Approach for ECG-Language Pretraining
par: Wang, Fuying, et autres
Publié: (2025)

Extreme Speech Classification in the Era of LLMs: Exploring Open-Source and Proprietary Models
par: Mahajan, Sarthak, et autres
Publié: (2025)

MIND: Math Informed syNthetic Dialogues for Pretraining LLMs
par: Akter, Syeda Nahida, et autres
Publié: (2024)

Mode-Conditioning Unlocks Superior Test-Time Scaling
par: Wu, Chen Henry, et autres
Publié: (2025)