Cover Image

Thinking Augmented Pre-training

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Wang, Liang, Yang, Nan, Huang, Shaohan, Dong, Li, Wei, Furu
Format:	Preprint
Published:	2025
Subjects:	Computation and Language Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Mixture of LoRA Experts
by: Wu, Xun, et al.
Published: (2024)

On-Policy RL with Optimal Reward Baseline
by: Hao, Yaru, et al.
Published: (2025)

Multi-Head Mixture-of-Experts
by: Wu, Xun, et al.
Published: (2024)

BitNet Distillation
by: Wu, Xun, et al.
Published: (2025)

Textual Aesthetics in Large Language Models
by: Jiang, Lingjie, et al.
Published: (2024)

The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
by: Ma, Shuming, et al.
Published: (2024)

Towards Stable and Effective Reinforcement Learning for Mixture-of-Experts
by: Zhang, Di, et al.
Published: (2025)

Multimodal Latent Language Modeling with Next-Token Diffusion
by: Sun, Yutao, et al.
Published: (2024)

Multiplex Thinking: Reasoning via Token-wise Branch-and-Merge
by: Tang, Yao, et al.
Published: (2026)

Breaking Training Bottlenecks: Effective and Stable Reinforcement Learning for Coding Models
by: Li, Zongqian, et al.
Published: (2026)

BitNet b1.58 2B4T Technical Report
by: Ma, Shuming, et al.
Published: (2025)

LongEmbed: Extending Embedding Models for Long Context Retrieval
by: Zhu, Dawei, et al.
Published: (2024)

PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training
by: Zhu, Dawei, et al.
Published: (2023)

Learning to Refine: Self-Refinement of Parallel Reasoning in LLMs
by: Wang, Qibin, et al.
Published: (2025)

Text Diffusion with Reinforced Conditioning
by: Liu, Yuxuan, et al.
Published: (2024)

MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning
by: Jiang, Ting, et al.
Published: (2024)

BitNet v2: Native 4-bit Activations with Hadamard Transformation for 1-bit LLMs
by: Wang, Hongyu, et al.
Published: (2025)

BitNet a4.8: 4-bit Activations for 1-bit LLMs
by: Wang, Hongyu, et al.
Published: (2024)

Chinese MentalBERT: Domain-Adaptive Pre-training on Social Media for Chinese Mental Health Text Analysis
by: Zhai, Wei, et al.
Published: (2024)

Differential Transformer
by: Ye, Tianzhu, et al.
Published: (2024)

Model Merging in Pre-training of Large Language Models
by: Li, Yunshui, et al.
Published: (2025)

Q-Sparse: All Large Language Models can be Fully Sparsely-Activated
by: Wang, Hongyu, et al.
Published: (2024)

Text Embeddings by Weakly-Supervised Contrastive Pre-training
by: Wang, Liang, et al.
Published: (2022)

Generative Representational Instruction Tuning
by: Muennighoff, Niklas, et al.
Published: (2024)

Scaling Data Difficulty: Improving Coding Models via Reinforcement Learning on Fresh and Challenging Problems
by: Li, Zongqian, et al.
Published: (2026)

Adapting Large Language Models to Domains via Reading Comprehension
by: Cheng, Daixuan, et al.
Published: (2023)

Enhancing One-shot Pruned Pre-trained Language Models through Sparse-Dense-Sparse Mechanism
by: Li, Guanchen, et al.
Published: (2024)

Using Pre-trained LLMs for Multivariate Time Series Forecasting
by: Wolff, Malcolm L., et al.
Published: (2025)

Bootstrapping Post-training Signals for Open-ended Tasks via Rubric-based Self-play on Pre-training Text
by: Huang, Chengyu, et al.
Published: (2026)

Auto-ICL: In-Context Learning without Human Supervision
by: Yang, Jinghan, et al.
Published: (2023)

HLAT: High-quality Large Language Model Pre-trained on AWS Trainium
by: Fan, Haozheng, et al.
Published: (2024)

Data Mixing Agent: Learning to Re-weight Domains for Continual Pre-training
by: Yang, Kailai, et al.
Published: (2025)

MLKD-BERT: Multi-level Knowledge Distillation for Pre-trained Language Models
by: Zhang, Ying, et al.
Published: (2024)

Instruction Pre-Training: Language Models are Supervised Multitask Learners
by: Cheng, Daixuan, et al.
Published: (2024)

DEPT: Decoupled Embeddings for Pre-training Language Models
by: Iacob, Alex, et al.
Published: (2024)

nanoLM: an Affordable LLM Pre-training Benchmark via Accurate Loss Prediction across Scales
by: Yao, Yiqun, et al.
Published: (2023)

Think-Augmented Function Calling: Improving LLM Parameter Accuracy Through Embedded Reasoning
by: Wei, Lei, et al.
Published: (2026)

Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training
by: Liu, Hong, et al.
Published: (2023)

On-Policy Context Distillation for Language Models
by: Ye, Tianzhu, et al.
Published: (2026)

Exploiting Vocabulary Frequency Imbalance in Language Model Pre-training
by: Chung, Woojin, et al.
Published: (2025)