Cover Image

Pre-Training Curriculum for Multi-Token Prediction in Language Models

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Aynetdinov, Ansar, Akbik, Alan
Format:	Preprint
Published:	2025
Subjects:	Computation and Language Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Repetition over Diversity: High-Signal Data Filtering for Sample-Efficient German Language Modeling
by: Aynetdinov, Ansar, et al.
Published: (2026)

NoiseBench: Benchmarking the Impact of Real Label Noise on Named Entity Recognition
by: Merdjanovska, Elena, et al.
Published: (2024)

SemScore: Automated Evaluation of Instruction-Tuned LLMs based on Semantic Textual Similarity
by: Aynetdinov, Ansar, et al.
Published: (2024)

Sample-Efficient Language Modeling with Linear Attention and Lightweight Enhancements
by: Haller, Patrick, et al.
Published: (2025)

What Matters in Linearizing Language Models? A Comparative Study of Architecture, Scale, and Task Adaptation
by: Haller, Patrick, et al.
Published: (2025)

Fabricator: An Open Source Toolkit for Generating Labeled Training Data with Teacher LLMs
by: Golde, Jonas, et al.
Published: (2023)

Pre-Trained Language Models for Keyphrase Prediction: A Review
by: Umair, Muhammad, et al.
Published: (2024)

Less is More: Pre-Training Cross-Lingual Small-Scale Language Models with Cognitively-Plausible Curriculum Learning Strategies
by: Salhan, Suchir, et al.
Published: (2024)

Evolution of Concepts in Language Model Pre-Training
by: Ge, Xuyang, et al.
Published: (2025)

Beyond Scaling: Measuring and Predicting the Upper Bound of Knowledge Retention in Language Model Pre-Training
by: Jiang, Changhao, et al.
Published: (2025)

Revealing the Inherent Instructability of Pre-Trained Language Models
by: An, Seokhyun, et al.
Published: (2024)

Beyond Next Token Prediction: Patch-Level Training for Large Language Models
by: Shao, Chenze, et al.
Published: (2024)

NEST-RQ: Next Token Prediction for Speech Self-Supervised Pre-Training
by: Han, Minglun, et al.
Published: (2024)

Token-Level Uncertainty-Aware Objective for Language Model Post-Training
by: Liu, Tingkai, et al.
Published: (2025)

Token Signature: Predicting Chain-of-Thought Gains with Token Decoding Feature in Large Language Models
by: Liu, Peijie, et al.
Published: (2025)

TinyHelen's First Curriculum: Training and Evaluating Tiny Language Models in a Simpler Language Environment
by: Yang, Ke, et al.
Published: (2024)

VocalNet-M2: Advancing Low-Latency Spoken Language Modeling via Integrated Multi-Codebook Tokenization and Multi-Token Prediction
by: Wang, Yuhao, et al.
Published: (2025)

Scaling LLM Pre-training with Vocabulary Curriculum
by: Yu, Fangyuan
Published: (2025)

TexIm FAST: Text-to-Image Representation for Semantic Similarity Evaluation using Transformers
by: Ansar, Wazib, et al.
Published: (2024)

From Transformers to LLMs: A Systematic Survey of Efficiency Considerations in NLP
by: Ansar, Wazib, et al.
Published: (2024)

Embedding-to-Prefix: Parameter-Efficient Personalization for Pre-Trained Large Language Models
by: Huber, Bernd, et al.
Published: (2025)

Legal Documents Drafting with Fine-Tuned Pre-Trained Large Language Model
by: Lin, Chun-Hsien, et al.
Published: (2024)

"According to ...": Prompting Language Models Improves Quoting from Pre-Training Data
by: Weller, Orion, et al.
Published: (2023)

A Study on Hidden Layer Distillation for Large Language Model Pre-Training
by: Guigon, Maxime, et al.
Published: (2026)

Unraveling Emotions with Pre-Trained Models
by: Pajón-Sanmartín, Alejandro, et al.
Published: (2025)

TEAL: Tokenize and Embed ALL for Multi-modal Large Language Models
by: Yang, Zhen, et al.
Published: (2023)

Multi-Stage Multi-Modal Pre-Training for Automatic Speech Recognition
by: Jain, Yash, et al.
Published: (2024)

Thinking Tokens for Language Modeling
by: Herel, David, et al.
Published: (2024)

ZeroTuning: Unlocking the Initial Token's Power to Enhance Large Language Models Without Training
by: Han, Feijiang, et al.
Published: (2025)

Construction of Hyper-Relational Knowledge Graphs Using Pre-Trained Large Language Models
by: Datta, Preetha, et al.
Published: (2024)

Training LLMs Beyond Next Token Prediction -- Filling the Mutual Information Gap
by: Yang, Chun-Hao, et al.
Published: (2025)

BabyHGRN: Exploring RNNs for Sample-Efficient Training of Language Models
by: Haller, Patrick, et al.
Published: (2024)

Training Text-to-Molecule Models with Context-Aware Tokenization
by: Kim, Seojin, et al.
Published: (2025)

Small Language Models for Curriculum-based Guidance
by: Katharakis, Konstantinos, et al.
Published: (2025)

Pre-Attention Expert Prediction and Prefetching for Mixture-of-Experts Large Language Models
by: Zhu, Shien, et al.
Published: (2025)

Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning
by: Xi, Zhiheng, et al.
Published: (2024)

Tokenization Matters! Degrading Large Language Models through Challenging Their Tokenization
by: Wang, Dixuan, et al.
Published: (2024)

SCRIPT: A Subcharacter Compositional Representation Injection Module for Korean Pre-Trained Language Models
by: Kim, SungHo, et al.
Published: (2026)

Amuro and Char: Analyzing the Relationship between Pre-Training and Fine-Tuning of Large Language Models
by: Sun, Kaiser, et al.
Published: (2024)

Untie the Knots: An Efficient Data Augmentation Strategy for Long-Context Pre-Training in Language Models
by: Tian, Junfeng, et al.
Published: (2024)