Image de couverture de livre

Sub-Scaling Laws: On the Role of Data Density and Training Strategies in LLMs

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Chen, Zhengyu, Wang, Siqi, Xiao, Teng, Wang, Yudong, Chen, Shiqi, Cai, Xunliang, He, Junxian, Wang, Jingang
Format:	Preprint
Publié:	2025
Sujets:	Machine Learning Artificial Intelligence
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Scaling and Transferability of Annealing Strategies in Large Language Model Training
par: Wang, Siqi, et autres
Publié: (2025)

Scaling Laws Across Model Architectures: A Comparative Analysis of Dense and MoE Models in Large Language Models
par: Wang, Siqi, et autres
Publié: (2024)

SpanNorm: Reconciling Training Stability and Performance in Deep Transformers
par: Wang, Chao, et autres
Publié: (2026)

What Makes Quantization for Large Language Models Hard? An Empirical Study from the Lens of Perturbation
par: Gong, Zhuocheng, et autres
Publié: (2024)

In-Context Sharpness as Alerts: An Inner Representation Perspective for Hallucination Mitigation
par: Chen, Shiqi, et autres
Publié: (2024)

How Do Your Code LLMs Perform? Empowering Code Instruction Tuning with High-Quality Data
par: Wang, Yejie, et autres
Publié: (2024)

Rethinking the Sampling Criteria in Reinforcement Learning for LLM Reasoning: A Competence-Difficulty Alignment Perspective
par: Kong, Deyang, et autres
Publié: (2025)

Preference Curriculum: LLMs Should Always Be Pretrained on Their Preferred Data
par: Zhang, Xuemiao, et autres
Publié: (2025)

Scaling Laws for Predicting Downstream Performance in LLMs
par: Chen, Yangyi, et autres
Publié: (2024)

P$^2$ Law: Scaling Law for Post-Training After Model Pruning
par: Chen, Xiaodong, et autres
Publié: (2024)

Reasoner for Real-World Event Detection: Scaling Reinforcement Learning via Adaptive Perplexity-Aware Sampling Strategy
par: Zhang, Xiaoyun, et autres
Publié: (2025)

MaskTab: Scalable Masked Tabular Pretraining with Scaling Laws and Distillation for Industrial Classification
par: Zheng, Bo, et autres
Publié: (2026)

How Should LLMs Consume High-Quality Data? Optimal Data Scheduling via Quality-Aware Functional Scaling Laws
par: Zhu, Zhitao, et autres
Publié: (2026)

DAQ: Density-Aware Post-Training Weight-Only Quantization For LLMs
par: Luo, Yingsong, et autres
Publié: (2024)

AutoScale: Scale-Aware Data Mixing for Pre-Training LLMs
par: Kang, Feiyang, et autres
Publié: (2024)

Understanding the Role of Training Data in Test-Time Scaling
par: Javanmard, Adel, et autres
Publié: (2025)

LLMs as Noisy Channels: A Shannon Perspective on Model Capacity and Scaling Laws
par: Ouyang, Xu, et autres
Publié: (2026)

HARMONIC: Harnessing LLMs for Tabular Data Synthesis and Privacy Protection
par: Wang, Yuxin, et autres
Publié: (2024)

Towards a Comprehensive Scaling Law of Mixture-of-Experts
par: Zhao, Guoliang, et autres
Publié: (2025)

LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws
par: Mayilvahanan, Prasanna, et autres
Publié: (2025)

Leveraging Invariant Principle for Heterophilic Graph Structure Distribution Shifts
par: Yang, Jinluan, et autres
Publié: (2024)

Using Scaling Laws for Data Source Utility Estimation in Domain-Specific Pre-Training
par: Ostapenko, Oleksiy, et autres
Publié: (2025)

Prefix Grouper: Efficient GRPO Training through Shared-Prefix Forward
par: Liu, Zikang, et autres
Publié: (2025)

Scaling Law with Learning Rate Annealing
par: Tissue, Howe, et autres
Publié: (2024)

Training LLMs for Honesty via Confessions
par: Joglekar, Manas, et autres
Publié: (2025)

CoScale-RL: Efficient Post-Training by Co-Scaling Data and Computation
par: Chen, Yutong, et autres
Publié: (2026)

SIRI: Self-Internalizing Reinforcement Learning with Intrinsic Skills for LLM Agent Training
par: He, Zhongyu, et autres
Publié: (2026)

Towards Off-Policy Reinforcement Learning for Ranking Policies with Human Feedback
par: Xiao, Teng, et autres
Publié: (2024)

AGRAG: Advanced Graph-based Retrieval-Augmented Generation for LLMs
par: Wang, Yubo, et autres
Publié: (2025)

Scaling Trends for Data Poisoning in LLMs
par: Bowen, Dillon, et autres
Publié: (2024)

Beyond Scaling Law: A Data-Efficient Distillation Framework for Reasoning
par: Wu, Xiaojun, et autres
Publié: (2025)

OptScale: Probabilistic Optimality for Inference-time Scaling
par: Wang, Youkang, et autres
Publié: (2025)

CoBA-RL: Capability-Oriented Budget Allocation for Reinforcement Learning in LLMs
par: Yao, Zhiyuan, et autres
Publié: (2026)

Ltri-LLM: Streaming Long Context Inference for LLMs with Training-Free Dynamic Triangular Attention Pattern
par: Tang, Hongyin, et autres
Publié: (2024)

Understanding LLM Behaviors via Compression: Data Generation, Knowledge Acquisition and Scaling Laws
par: Pan, Zhixuan, et autres
Publié: (2025)

Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs
par: Bian, Song, et autres
Publié: (2025)

MaskPrune: Mask-based LLM Pruning for Layer-wise Uniform Structures
par: Qin, Jiayu, et autres
Publié: (2025)

How to Train Data-Efficient LLMs
par: Sachdeva, Noveen, et autres
Publié: (2024)

SparseBalance: Load-Balanced Long Context Training with Dynamic Sparse Attention
par: Xu, Hongtao, et autres
Publié: (2026)

Towards Neural Scaling Laws on Graphs
par: Liu, Jingzhe, et autres
Publié: (2024)