Imagen de Portada

Efficient Pretraining Length Scaling

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Wu, Bohong, Yan, Shen, Zhang, Sijun, Lu, Jianqiao, Zeng, Yutao, Wang, Ya, Zhou, Xun
Formato:	Preprint
Publicado:	2025
Materias:	Computation and Language
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid Normalization
por: Zhuo, Zhijian, et al.
Publicado: (2025)

Scale-Distribution Decoupling: Enabling Stable and Effective Training of Large Language Models
por: Wang, Ya, et al.
Publicado: (2025)

Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling
por: Huang, Hongzhi, et al.
Publicado: (2025)

Parallel Loop Transformer for Efficient Test-Time Computation Scaling
por: Wu, Bohong, et al.
Publicado: (2025)

FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inference
por: Lai, Xunhao, et al.
Publicado: (2025)

HRM-Text: Efficient Pretraining Beyond Scaling
por: Wang, Guan, et al.
Publicado: (2026)

Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Length
por: Ma, Xuezhe, et al.
Publicado: (2024)

Polynomial Composition Activations: Unleashing the Dynamics of Large Language Models
por: Zhuo, Zhijian, et al.
Publicado: (2024)

Seeing the Image: Prioritizing Visual Correlation by Contrastive Alignment
por: Xiao, Xin, et al.
Publicado: (2024)

Length Value Model: Scalable Value Pretraining for Token-Level Length Modeling
por: Zhang, Zhen, et al.
Publicado: (2026)

Universal YOCO for Efficient Depth Scaling
por: Sun, Yutao, et al.
Publicado: (2026)

MachineLearningLM: Scaling Many-shot In-context Learning via Continued Pretraining
por: Dong, Haoyu, et al.
Publicado: (2025)

Scaling Law for Quantization-Aware Training
por: Chen, Mengzhao, et al.
Publicado: (2025)

Anti-Length Shift: Dynamic Outlier Truncation for Training Efficient Reasoning Models
por: Wu, Wei, et al.
Publicado: (2026)

Clustering Algorithms and RAG Enhancing Semi-Supervised Text Classification with Large LLMs
por: Zhong, Shan, et al.
Publicado: (2024)

QuaDMix: Quality-Diversity Balanced Data Selection for Efficient LLM Pretraining
por: Liu, Fengze, et al.
Publicado: (2025)

PhenoLIP: Integrating Phenotype Ontology Knowledge into Medical Vision-Language Pretraining
por: Liang, Cheng, et al.
Publicado: (2026)

Scaling Laws For Mixed Quantization
por: Cao, Zeyu, et al.
Publicado: (2024)

Frac-Connections: Fractional Extension of Hyper-Connections
por: Zhu, Defa, et al.
Publicado: (2025)

An Integrated Data Processing Framework for Pretraining Foundation Models
por: Sun, Yiding, et al.
Publicado: (2024)

Beyond Length: Quantifying Long-Range Information for Long-Context LLM Pretraining Data
por: Deng, Haoran, et al.
Publicado: (2025)

AttentionInfluence: Adopting Attention Head Influence for Weak-to-Strong Pretraining Data Selection
por: Hua, Kai, et al.
Publicado: (2025)

Decoupling Safety into Orthogonal Subspace: Cost-Efficient and Performance-Preserving Alignment for Large Language Models
por: Mou, Yutao, et al.
Publicado: (2025)

Flora: Effortless Context Construction to Arbitrary Length and Scale
por: Chen, Tianxiang, et al.
Publicado: (2025)

LongSkywork: A Training Recipe for Efficiently Extending Context Length in Large Language Models
por: Zhao, Liang, et al.
Publicado: (2024)

Reformulation for Pretraining Data Augmentation
por: Hao, Xintong, et al.
Publicado: (2025)

WRAP++: Web discoveRy Amplified Pretraining
por: Zhou, Jiang, et al.
Publicado: (2026)

DCIS: Efficient Length Extrapolation of LLMs via Divide-and-Conquer Scaling Factor Search
por: Yang, Lei, et al.
Publicado: (2024)

Length Generalization of Causal Transformers without Position Encoding
por: Wang, Jie, et al.
Publicado: (2024)

BIDER: Bridging Knowledge Inconsistency for Efficient Retrieval-Augmented LLMs via Key Supporting Evidence
por: Jin, Jiajie, et al.
Publicado: (2024)

Pretraining A Large Language Model using Distributed GPUs: A Memory-Efficient Decentralized Paradigm
por: Zhang, Jinrui, et al.
Publicado: (2026)

Various Lengths, Constant Speed: Efficient Language Modeling with Lightning Attention
por: Qin, Zhen, et al.
Publicado: (2024)

LAPO: Internalizing Reasoning Efficiency via Length-Adaptive Policy Optimization
por: Wu, Xingyu, et al.
Publicado: (2025)

Hyper-Connections
por: Zhu, Defa, et al.
Publicado: (2024)

SimReg: Achieving Higher Performance in the Pretraining via Embedding Similarity Regularization
por: Sun, Yan, et al.
Publicado: (2026)

Accurate Scene Text Recognition with Efficient Model Scaling and Cloze Self-Distillation
por: Maracani, Andrea, et al.
Publicado: (2025)

PretrainZero: Reinforcement Active Pretraining
por: Xing, Xingrun, et al.
Publicado: (2025)

Language Models and Cycle Consistency for Self-Reflective Machine Translation
por: Wangni, Jianqiao
Publicado: (2024)

Efficient Pretraining Data Selection for Language Models via Multi-Actor Collaboration
por: Bai, Tianyi, et al.
Publicado: (2024)

Budget-aware Test-time Scaling via Discriminative Verification
por: Montgomery, Kyle, et al.
Publicado: (2025)