Copertina

Joint Selection for Large-Scale Pre-Training Data via Policy Gradient-based Mask Learning

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Fan, Ziqing, Xian, Yuqiao, Sun, Yan, Shen, Li
Natura:	Preprint
Pubblicazione:	2025
Soggetti:	Computation and Language Machine Learning
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Joint Training for Selective Prediction
di: Li, Zhaohui, et al.
Pubblicazione: (2024)

AutoScale: Scale-Aware Data Mixing for Pre-Training LLMs
di: Kang, Feiyang, et al.
Pubblicazione: (2024)

Reinforcement Learning on Pre-Training Data
di: Li, Siheng, et al.
Pubblicazione: (2025)

Scaling Multi-Hop Training Data via Graph-Constrained Path Selection
di: Chen, Pengyu, et al.
Pubblicazione: (2026)

Min-K%++: Improved Baseline for Detecting Pre-Training Data from Large Language Models
di: Zhang, Jingyang, et al.
Pubblicazione: (2024)

FGGM: Fisher-Guided Gradient Masking for Continual Learning
di: Tan, Chao-Hong, et al.
Pubblicazione: (2026)

Centered Masking for Language-Image Pre-Training
di: Liang, Mingliang, et al.
Pubblicazione: (2024)

Synthetic Text Generation for Training Large Language Models via Gradient Matching
di: Nguyen, Dang, et al.
Pubblicazione: (2025)

Bypass Back-propagation: Optimization-based Structural Pruning for Large Language Models via Policy Gradient
di: Gao, Yuan, et al.
Pubblicazione: (2024)

FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale
di: Patel, Ajay, et al.
Pubblicazione: (2026)

Pre-Trained Policy Discriminators are General Reward Models
di: Dou, Shihan, et al.
Pubblicazione: (2025)

Revisiting Replay and Gradient Alignment for Continual Pre-Training of Large Language Models
di: Abbes, Istabrak, et al.
Pubblicazione: (2025)

Influential Language Data Selection via Gradient Trajectory Pursuit
di: Deng, Zhiwei, et al.
Pubblicazione: (2024)

Dialogue for Prompting: a Policy-Gradient-Based Discrete Prompt Generation for Few-shot Learning
di: Li, Chengzhengxu, et al.
Pubblicazione: (2023)

GradAlign: Gradient-Aligned Data Selection for LLM Reinforcement Learning
di: Yang, Ningyuan, et al.
Pubblicazione: (2026)

Efficient Data Selection at Scale via Influence Distillation
di: Nikdan, Mahdi, et al.
Pubblicazione: (2025)

Learning Dynamics in Continual Pre-Training for Large Language Models
di: Wang, Xingjin, et al.
Pubblicazione: (2025)

Intercept Cancer: Cancer Pre-Screening with Large Scale Healthcare Foundation Models
di: Sun, Liwen, et al.
Pubblicazione: (2025)

Joint Unsupervised and Supervised Training for Automatic Speech Recognition via Bilevel Optimization
di: Saif, A F M, et al.
Pubblicazione: (2024)

CE-GPPO: Coordinating Entropy via Gradient-Preserving Clipping Policy Optimization in Reinforcement Learning
di: Su, Zhenpeng, et al.
Pubblicazione: (2025)

Combatting Dimensional Collapse in LLM Pre-Training Data via Diversified File Selection
di: Fan, Ziqing, et al.
Pubblicazione: (2025)

AGGC: Adaptive Group Gradient Clipping for Stabilizing Large Language Model Training
di: Li, Zhiyuan, et al.
Pubblicazione: (2026)

GPAS: Accelerating Convergence of LLM Pretraining via Gradient-Preserving Activation Scaling
di: Chen, Tianhao, et al.
Pubblicazione: (2025)

Efficient RLVR Training via Weighted Mutual Information Data Selection
di: Zhou, Xinyu, et al.
Pubblicazione: (2026)

Optimizing Pre-Training Data Mixtures with Mixtures of Data Expert Models
di: Belenki, Lior, et al.
Pubblicazione: (2025)

Perplexity-Aware Data Scaling Law: Perplexity Landscapes Predict Performance for Continual Pre-training
di: Liu, Lei, et al.
Pubblicazione: (2025)

Time-Aware Feature Selection: Adaptive Temporal Masking for Stable Sparse Autoencoder Training
di: Li, T. Ed, et al.
Pubblicazione: (2025)

Automated Text Mining of Experimental Methodologies from Biomedical Literature
di: Guo, Ziqing
Pubblicazione: (2024)

Programming Every Example: Lifting Pre-training Data Quality Like Experts at Scale
di: Zhou, Fan, et al.
Pubblicazione: (2024)

SVFit: Parameter-Efficient Fine-Tuning of Large Pre-Trained Models Using Singular Values
di: Sun, Chengwei, et al.
Pubblicazione: (2024)

Scaling Law for Language Models Training Considering Batch Size
di: Shuai, Xian, et al.
Pubblicazione: (2024)

Scaling up Masked Diffusion Models on Text
di: Nie, Shen, et al.
Pubblicazione: (2024)

Dynamic Gradient Alignment for Online Data Mixing
di: Fan, Simin, et al.
Pubblicazione: (2024)

Linear-Time Demonstration Selection for In-Context Learning via Gradient Estimation
di: Zhang, Ziniu, et al.
Pubblicazione: (2025)

Mask-Enhanced Autoregressive Prediction: Pay Less Attention to Learn More
di: Zhuang, Xialie, et al.
Pubblicazione: (2025)

Development of Pre-Trained Transformer-based Models for the Nepali Language
di: Thapa, Prajwal, et al.
Pubblicazione: (2024)

Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learning
di: Zhao, Shiwan, et al.
Pubblicazione: (2026)

Prescriptive Scaling Laws for Data Constrained Training
di: Lovelace, Justin, et al.
Pubblicazione: (2026)

MedFLIP: Medical Vision-and-Language Self-supervised Fast Pre-Training with Masked Autoencoder
di: Li, Lei, et al.
Pubblicazione: (2024)

From $P(y|x)$ to $P(y)$: Investigating Reinforcement Learning in Pre-train Space
di: Tan, Yuqiao, et al.
Pubblicazione: (2026)