Imagen de Portada

Joint Selection for Large-Scale Pre-Training Data via Policy Gradient-based Mask Learning

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Fan, Ziqing, Xian, Yuqiao, Sun, Yan, Shen, Li
Formato:	Preprint
Publicado:	2025
Materias:	Computation and Language Machine Learning
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Joint Training for Selective Prediction
por: Li, Zhaohui, et al.
Publicado: (2024)

AutoScale: Scale-Aware Data Mixing for Pre-Training LLMs
por: Kang, Feiyang, et al.
Publicado: (2024)

Reinforcement Learning on Pre-Training Data
por: Li, Siheng, et al.
Publicado: (2025)

Scaling Multi-Hop Training Data via Graph-Constrained Path Selection
por: Chen, Pengyu, et al.
Publicado: (2026)

Min-K%++: Improved Baseline for Detecting Pre-Training Data from Large Language Models
por: Zhang, Jingyang, et al.
Publicado: (2024)

FGGM: Fisher-Guided Gradient Masking for Continual Learning
por: Tan, Chao-Hong, et al.
Publicado: (2026)

Centered Masking for Language-Image Pre-Training
por: Liang, Mingliang, et al.
Publicado: (2024)

Synthetic Text Generation for Training Large Language Models via Gradient Matching
por: Nguyen, Dang, et al.
Publicado: (2025)

Bypass Back-propagation: Optimization-based Structural Pruning for Large Language Models via Policy Gradient
por: Gao, Yuan, et al.
Publicado: (2024)

FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale
por: Patel, Ajay, et al.
Publicado: (2026)

Pre-Trained Policy Discriminators are General Reward Models
por: Dou, Shihan, et al.
Publicado: (2025)

Revisiting Replay and Gradient Alignment for Continual Pre-Training of Large Language Models
por: Abbes, Istabrak, et al.
Publicado: (2025)

Influential Language Data Selection via Gradient Trajectory Pursuit
por: Deng, Zhiwei, et al.
Publicado: (2024)

Dialogue for Prompting: a Policy-Gradient-Based Discrete Prompt Generation for Few-shot Learning
por: Li, Chengzhengxu, et al.
Publicado: (2023)

GradAlign: Gradient-Aligned Data Selection for LLM Reinforcement Learning
por: Yang, Ningyuan, et al.
Publicado: (2026)

Efficient Data Selection at Scale via Influence Distillation
por: Nikdan, Mahdi, et al.
Publicado: (2025)

Learning Dynamics in Continual Pre-Training for Large Language Models
por: Wang, Xingjin, et al.
Publicado: (2025)

Intercept Cancer: Cancer Pre-Screening with Large Scale Healthcare Foundation Models
por: Sun, Liwen, et al.
Publicado: (2025)

Joint Unsupervised and Supervised Training for Automatic Speech Recognition via Bilevel Optimization
por: Saif, A F M, et al.
Publicado: (2024)

CE-GPPO: Coordinating Entropy via Gradient-Preserving Clipping Policy Optimization in Reinforcement Learning
por: Su, Zhenpeng, et al.
Publicado: (2025)

Combatting Dimensional Collapse in LLM Pre-Training Data via Diversified File Selection
por: Fan, Ziqing, et al.
Publicado: (2025)

AGGC: Adaptive Group Gradient Clipping for Stabilizing Large Language Model Training
por: Li, Zhiyuan, et al.
Publicado: (2026)

GPAS: Accelerating Convergence of LLM Pretraining via Gradient-Preserving Activation Scaling
por: Chen, Tianhao, et al.
Publicado: (2025)

Efficient RLVR Training via Weighted Mutual Information Data Selection
por: Zhou, Xinyu, et al.
Publicado: (2026)

Optimizing Pre-Training Data Mixtures with Mixtures of Data Expert Models
por: Belenki, Lior, et al.
Publicado: (2025)

Perplexity-Aware Data Scaling Law: Perplexity Landscapes Predict Performance for Continual Pre-training
por: Liu, Lei, et al.
Publicado: (2025)

Time-Aware Feature Selection: Adaptive Temporal Masking for Stable Sparse Autoencoder Training
por: Li, T. Ed, et al.
Publicado: (2025)

Automated Text Mining of Experimental Methodologies from Biomedical Literature
por: Guo, Ziqing
Publicado: (2024)

Programming Every Example: Lifting Pre-training Data Quality Like Experts at Scale
por: Zhou, Fan, et al.
Publicado: (2024)

SVFit: Parameter-Efficient Fine-Tuning of Large Pre-Trained Models Using Singular Values
por: Sun, Chengwei, et al.
Publicado: (2024)

Scaling Law for Language Models Training Considering Batch Size
por: Shuai, Xian, et al.
Publicado: (2024)

Scaling up Masked Diffusion Models on Text
por: Nie, Shen, et al.
Publicado: (2024)

Dynamic Gradient Alignment for Online Data Mixing
por: Fan, Simin, et al.
Publicado: (2024)

Linear-Time Demonstration Selection for In-Context Learning via Gradient Estimation
por: Zhang, Ziniu, et al.
Publicado: (2025)

Mask-Enhanced Autoregressive Prediction: Pay Less Attention to Learn More
por: Zhuang, Xialie, et al.
Publicado: (2025)

Development of Pre-Trained Transformer-based Models for the Nepali Language
por: Thapa, Prajwal, et al.
Publicado: (2024)

Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learning
por: Zhao, Shiwan, et al.
Publicado: (2026)

Prescriptive Scaling Laws for Data Constrained Training
por: Lovelace, Justin, et al.
Publicado: (2026)

MedFLIP: Medical Vision-and-Language Self-supervised Fast Pre-Training with Masked Autoencoder
por: Li, Lei, et al.
Publicado: (2024)

From $P(y|x)$ to $P(y)$: Investigating Reinforcement Learning in Pre-train Space
por: Tan, Yuqiao, et al.
Publicado: (2026)