Cover Image

LLM Pretraining with Continuous Concepts

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Tack, Jihoon, Lanchantin, Jack, Yu, Jane, Cohen, Andrew, Kulikov, Ilia, Lan, Janice, Hao, Shibo, Tian, Yuandong, Weston, Jason, Li, Xian
Format:	Preprint
Published:	2025
Subjects:	Machine Learning Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

OptimalThinkingBench: Evaluating Over and Underthinking in LLMs
by: Aggarwal, Pranjal, et al.
Published: (2025)

Self-Improving Pretraining: using post-trained models to pretrain better models
by: Tan, Ellen Xiaoqing, et al.
Published: (2026)

Adaptive Decoding via Latent Preference Optimization
by: Dhuliawala, Shehzaad, et al.
Published: (2024)

Diverse Preference Optimization
by: Lanchantin, Jack, et al.
Published: (2025)

J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning
by: Whitehouse, Chenxi, et al.
Published: (2025)

Bridging Offline and Online Reinforcement Learning for LLMs
by: Lanchantin, Jack, et al.
Published: (2025)

Training Large Language Models to Reason in a Continuous Latent Space
by: Hao, Shibo, et al.
Published: (2024)

SPICE: Self-Play In Corpus Environments Improves Reasoning
by: Liu, Bo, et al.
Published: (2025)

CoT-Self-Instruct: Building high-quality synthetic prompts for reasoning and non-reasoning tasks
by: Yu, Ping, et al.
Published: (2025)

ReMoDetect: Reward Models Recognize Aligned LLM's Generations
by: Lee, Hyunseok, et al.
Published: (2024)

Distilling System 2 into System 1
by: Yu, Ping, et al.
Published: (2024)

Jointly Reinforcing Diversity and Quality in Language Model Generations
by: Li, Tianjian, et al.
Published: (2025)

NaturalReasoning: Reasoning in the Wild with 2.8M Challenging Questions
by: Yuan, Weizhe, et al.
Published: (2025)

NaturalThoughts: Selecting and Distilling Reasoning Traces for General Reasoning Tasks
by: Li, Yang, et al.
Published: (2025)

TOOLVERIFIER: Generalization to New Tools via Self-Verification
by: Mekala, Dheeraj, et al.
Published: (2024)

Hybrid Reinforcement: When Reward Is Sparse, It's Better to Be Dense
by: Tao, Leitian, et al.
Published: (2025)

Beyond Correctness: Learning Robust Reasoning via Transfer
by: Lee, Hyunseok, et al.
Published: (2026)

The Majority is not always right: RL training for solution aggregation
by: Zhao, Wenting, et al.
Published: (2025)

ReVISE: Learning to Refine at Test-Time via Intrinsic Self-Verification
by: Lee, Hyunseok, et al.
Published: (2025)

SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Tasks
by: Zhou, Yifei, et al.
Published: (2025)

Following Length Constraints in Instructions
by: Yuan, Weizhe, et al.
Published: (2024)

Composing Global Solutions to Reasoning Tasks via Algebraic Objects in Neural Nets
by: Tian, Yuandong
Published: (2024)

Post-training an LLM for RAG? Train on Self-Generated Demonstrations
by: Finlayson, Matthew, et al.
Published: (2025)

Self-Taught Evaluators
by: Wang, Tianlu, et al.
Published: (2024)

ALMA: Alignment with Minimal Annotation
by: Yasunaga, Michihiro, et al.
Published: (2024)

Online Adaptation of Language Models with a Memory of Amortized Contexts
by: Tack, Jihoon, et al.
Published: (2024)

Reasoning by Superposition: A Theoretical Perspective on Chain of Continuous Thought
by: Zhu, Hanlin, et al.
Published: (2025)

Emergence of Superposition: Unveiling the Training Dynamics of Chain of Continuous Thought
by: Zhu, Hanlin, et al.
Published: (2025)

Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge
by: Wu, Tianhao, et al.
Published: (2024)

Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Length
by: Ma, Xuezhe, et al.
Published: (2024)

AdvPrefix: An Objective for Nuanced LLM Jailbreaks
by: Zhu, Sicheng, et al.
Published: (2024)

Branch-Solve-Merge Improves Large Language Model Evaluation and Generation
by: Saha, Swarnadeep, et al.
Published: (2023)

TiC-LM: A Web-Scale Benchmark for Time-Continual LLM Pretraining
by: Li, Jeffrey, et al.
Published: (2025)

LoCoCo: Dropping In Convolutions for Long Context Compression
by: Cai, Ruisi, et al.
Published: (2024)

Mining Hidden Thoughts from Texts: Evaluating Continual Pretraining with Synthetic Data for LLM Reasoning
by: Ishibashi, Yoichi, et al.
Published: (2025)

Learning to Reason for Factuality
by: Chen, Xilun, et al.
Published: (2025)

Thinking LLMs: General Instruction Following with Thought Generation
by: Wu, Tianhao, et al.
Published: (2024)

Dissecting Outlier Dynamics in LLM NVFP4 Pretraining
by: Dong, Peijie, et al.
Published: (2026)

Model Medicine: A Clinical Framework for Understanding, Diagnosing, and Treating AI Models
by: Jeong, Jihoon
Published: (2026)

Better Alignment with Instruction Back-and-Forth Translation
by: Nguyen, Thao, et al.
Published: (2024)