Cover Image

Beyond Random Sampling: Efficient Language Model Pretraining via Curriculum Learning

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Zhang, Yang, Mohamed, Amr, Abdine, Hadi, Shang, Guokan, Vazirgiannis, Michalis
Format:	Preprint
Published:	2025
Subjects:	Computation and Language Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

LLM as a Broken Telephone: Iterative Generation Distorts Information
by: Mohamed, Amr, et al.
Published: (2025)

Markovian Generation Chains in Large Language Models
by: Geng, Mingmeng, et al.
Published: (2026)

Nile-Chat: Egyptian Language Models for Arabic and Latin Scripts
by: Shang, Guokan, et al.
Published: (2025)

Fast-Decoding Diffusion Language Models via Progress-Aware Confidence Schedules
by: Mohamed, Amr, et al.
Published: (2025)

Shorter but not Worse: Frugal Reasoning via Easy Samples as Length Regularizers in Math RLVR
by: Bounhar, Abdelaziz, et al.
Published: (2025)

Leveraging Discourse Structure for Extractive Meeting Summarization
by: Rennard, Virgile, et al.
Published: (2024)

Lost in the Mix: Evaluating LLM Understanding of Code-Switched Text
by: Mohamed, Amr, et al.
Published: (2025)

YaPO: Learnable Sparse Activation Steering Vectors for Domain Adaptation
by: Bounhar, Abdelaziz, et al.
Published: (2026)

The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Text
by: Guo, Yanzhu, et al.
Published: (2023)

Graph Linearization Methods for Reasoning on Graphs with Large Language Models
by: Xypolopoulos, Christos, et al.
Published: (2024)

Atlas-Chat: Adapting Large Language Models for Low-Resource Moroccan Arabic Dialect
by: Shang, Guokan, et al.
Published: (2024)

Word Sense Induction with Hierarchical Clustering and Mutual Information Maximization
by: Abdine, Hadi, et al.
Published: (2022)

Prot2Text: Multimodal Protein's Function Generation with GNNs and Transformers
by: Abdine, Hadi, et al.
Published: (2023)

Bias in the Mirror: Are LLMs opinions robust to their own adversarial attacks ?
by: Rennard, Virgile, et al.
Published: (2024)

Beyond Repetition: Text Simplification and Curriculum Learning for Data-Constrained Pretraining
by: Roque, Matthew Theodore, et al.
Published: (2025)

GreekMMLU: A Native-Sourced Multitask Benchmark for Evaluating Language Models in Greek
by: Zhang, Yang, et al.
Published: (2026)

Explaining Predictions by Characteristic Rules
by: Alkhatib, Amr, et al.
Published: (2024)

An Evaluation of Large Language Models on Text Summarization Tasks Using Prompt Engineering Techniques
by: Aly, Walid Mohamed, et al.
Published: (2025)

Preference Curriculum: LLMs Should Always Be Pretrained on Their Preferred Data
by: Zhang, Xuemiao, et al.
Published: (2025)

In-context Pretraining: Language Modeling Beyond Document Boundaries
by: Shi, Weijia, et al.
Published: (2023)

Curriculum Learning for LLM Pretraining: An Analysis of Learning Dynamics
by: Elgaar, Mohamed, et al.
Published: (2026)

Accelerating Large Language Model Pretraining via LFR Pedagogy: Learn, Focus, and Review
by: Prakriya, Neha, et al.
Published: (2024)

Automated Rubrics for Reliable Evaluation of Medical Dialogue Systems
by: Chen, Yinzhu, et al.
Published: (2026)

A Knowledge-Injected Curriculum Pretraining Framework for Question Answering
by: Lin, Xin, et al.
Published: (2024)

Text Generation Beyond Discrete Token Sampling
by: Zhuang, Yufan, et al.
Published: (2025)

MASS: Mathematical Data Selection via Skill Graphs for Pretraining Large Language Models
by: Li, Jiazheng, et al.
Published: (2025)

Systematic Analysis for Pretrained Language Model Priming for Parameter-Efficient Fine-tuning
by: Huang, Shih-Cheng, et al.
Published: (2022)

Beyond Memorization: The Challenge of Random Memory Access in Language Models
by: Zhu, Tongyao, et al.
Published: (2024)

Infi-MMR: Curriculum-based Unlocking Multimodal Reasoning via Phased Reinforcement Learning in Multimodal Small Language Models
by: Liu, Zeyu, et al.
Published: (2025)

Multilingual Pretraining for Pixel Language Models
by: Kesen, Ilker, et al.
Published: (2025)

Fisher Information-based Efficient Curriculum Federated Learning with Large Language Models
by: Liu, Ji, et al.
Published: (2024)

Beyond URLs: Metadata Diversity and Position for Efficient LLM Pretraining
by: Fan, Dongyang, et al.
Published: (2025)

Persian-Phi: Efficient Cross-Lingual Adaptation of Compact LLMs via Curriculum Learning
by: Akhlaghi, Amir Mohammad, et al.
Published: (2025)

Linguistic Blind Spots in Clinical Decision Extraction
by: Elgaar, Mohamed, et al.
Published: (2026)

Where to Begin: Efficient Pretraining via Subnetwork Selection and Distillation
by: Krishnakumar, Arjun, et al.
Published: (2025)

Interpretable Graph Neural Networks for Tabular Data
by: Alkhatib, Amr, et al.
Published: (2023)

TACLer: Tailored Curriculum Reinforcement Learning for Efficient Reasoning
by: Lai, Huiyuan, et al.
Published: (2026)

CodePMP: Scalable Preference Model Pretraining for Large Language Model Reasoning
by: Yu, Huimu, et al.
Published: (2024)

Exploring Cultural Variations in Moral Judgments with Large Language Models
by: Mohammadi, Hadi, et al.
Published: (2025)

A Severity-Based Curriculum Learning Strategy for Arabic Medical Text Generation
by: Alansary, Ahmed, et al.
Published: (2026)