Cover Image

PretrainZero: Reinforcement Active Pretraining

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Xing, Xingrun, Fan, Zhiyuan, Lou, Jie, Li, Guoqi, Zhang, Jiajun, Zhang, Debing
Format:	Preprint
Published:	2025
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

CodePMP: Scalable Preference Model Pretraining for Large Language Model Reasoning
by: Yu, Huimu, et al.
Published: (2024)

SpikeLM: Towards General Spike-Driven Language Modeling via Elastic Bi-Spiking Mechanisms
by: Xing, Xingrun, et al.
Published: (2024)

SpikeLLM: Scaling up Spiking Neural Network to Large Language Models via Saliency-based Spiking
by: Xing, Xingrun, et al.
Published: (2024)

Tackling Length Inflation Without Trade-offs: Group Relative Reward Rescaling for Reinforcement Learning
by: Li, Zichao, et al.
Published: (2026)

EfficientLLM: Scalable Pruning-Aware Pretraining for Architecture-Agnostic Edge Language Models
by: Xing, Xingrun, et al.
Published: (2025)

Reformulation for Pretraining Data Augmentation
by: Hao, Xintong, et al.
Published: (2025)

Craw4LLM: Efficient Web Crawling for LLM Pretraining
by: Yu, Shi, et al.
Published: (2025)

WRAP++: Web discoveRy Amplified Pretraining
by: Zhou, Jiang, et al.
Published: (2026)

Cleaner Pretraining Corpus Curation with Neural Web Scraping
by: Xu, Zhipeng, et al.
Published: (2024)

Efficient Pretraining Length Scaling
by: Wu, Bohong, et al.
Published: (2025)

RLP: Reinforcement as a Pretraining Objective
by: Hatamizadeh, Ali, et al.
Published: (2025)

Improving Language Plasticity via Pretraining with Active Forgetting
by: Chen, Yihong, et al.
Published: (2023)

Multilingual Pretraining and Instruction Tuning Improve Cross-Lingual Knowledge Alignment, But Only Shallowly
by: Gao, Changjiang, et al.
Published: (2024)

ARWKV: Pretrain is not what we need, an RNN-Attention-Based Language Model Born from Transformer
by: Yueyu, Lin, et al.
Published: (2025)

Towards Efficient Active Learning in NLP via Pretrained Representations
by: Vysogorets, Artem, et al.
Published: (2024)

Vision-and-Language Pretraining
by: Nguyen, Thong, et al.
Published: (2022)

MemSearcher: Training LLMs to Reason, Search and Manage Memory via End-to-End Reinforcement Learning
by: Yuan, Qianhao, et al.
Published: (2025)

Coupled Variational Reinforcement Learning for Language Model General Reasoning
by: Wen, Xueru, et al.
Published: (2025)

KLoB: a Benchmark for Assessing Knowledge Locating Methods in Language Models
by: Ju, Yiming, et al.
Published: (2023)

ReactXT: Understanding Molecular "Reaction-ship" via Reaction-Contextualized Molecule-Text Pretraining
by: Liu, Zhiyuan, et al.
Published: (2024)

Don't Throw Away Your Pretrained Model
by: Feng, Shangbin, et al.
Published: (2025)

Model-Generated Pretraining Signals Improves Zero-Shot Generalization of Text-to-Text Transformers
by: Gong, Linyuan, et al.
Published: (2023)

Dissecting Outlier Dynamics in LLM NVFP4 Pretraining
by: Dong, Peijie, et al.
Published: (2026)

Pretraining Finnish ModernBERTs
by: Reunamo, Akseli, et al.
Published: (2025)

Evaluating Unsupervised Dimensionality Reduction Methods for Pretrained Sentence Embeddings
by: Zhang, Gaifan, et al.
Published: (2024)

A Systematic Survey of Semantic Role Labeling in the Era of Pretrained Language Models
by: Chen, Huiyao, et al.
Published: (2025)

On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Survey
by: Zhang, Meishan, et al.
Published: (2025)

Group-Level Data Selection for Efficient Pretraining
by: Yu, Zichun, et al.
Published: (2025)

Pretraining Data Detection for Large Language Models: A Divergence-based Calibration Method
by: Zhang, Weichao, et al.
Published: (2024)

LLM Pretraining with Continuous Concepts
by: Tack, Jihoon, et al.
Published: (2025)

PRESTO: Progressive Pretraining Enhances Synthetic Chemistry Outcomes
by: Cao, He, et al.
Published: (2024)

Exploring Pretraining via Active Forgetting for Improving Cross Lingual Transfer for Decoder Language Models
by: Aggarwal, Divyanshu, et al.
Published: (2024)

Adapting Pretrained Language Models for Citation Classification via Self-Supervised Contrastive Learning
by: Li, Tong, et al.
Published: (2025)

Scalable Oversight for Superhuman AI via Recursive Self-Critiquing
by: Wen, Xueru, et al.
Published: (2025)

GPR: Empowering Generation with Graph-Pretrained Retriever
by: Wang, Xiaochen, et al.
Published: (2025)

Position-Aware Depth Decay Decoding ($D^3$): Boosting Large Language Model Inference Efficiency
by: Fan, Siqi, et al.
Published: (2025)

Midtraining Bridges Pretraining and Posttraining Distributions
by: Liu, Emmy, et al.
Published: (2025)

Pretraining Language Models Using Translationese
by: Doshi, Meet, et al.
Published: (2024)

Pretraining and Benchmarking Modern Encoders for Latvian
by: Znotins, Arturs
Published: (2026)

Geographic Adaptation of Pretrained Language Models
by: Hofmann, Valentin, et al.
Published: (2022)