Image de couverture de livre

Pre-training under infinite compute

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Kim, Konwoo, Kotha, Suhas, Liang, Percy, Hashimoto, Tatsunori
Format:	Preprint
Publié:	2025
Sujets:	Machine Learning
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Data-efficient pre-training by scaling synthetic megadocs
par: Kim, Konwoo, et autres
Publié: (2026)

Replaying pre-training data improves fine-tuning
par: Kotha, Suhas, et autres
Publié: (2026)

Evaluating Self-Supervised Learning via Risk Decomposition
par: Dubois, Yann, et autres
Publié: (2023)

Synthetic Mixed Training: Scaling Parametric Knowledge Acquisition Beyond RAG
par: Han, Seungju, et autres
Publié: (2026)

Robust Distortion-free Watermarks for Language Models
par: Kuditipudi, Rohith, et autres
Publié: (2023)

Testing the Limits of Jailbreaking Defenses with the Purple Problem
par: Kim, Taeyoun, et autres
Publié: (2024)

On the Learnability of Watermarks for Language Models
par: Gu, Chenchen, et autres
Publié: (2023)

Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators
par: Dubois, Yann, et autres
Publié: (2024)

Understanding Catastrophic Forgetting in Language Models via Implicit Inference
par: Kotha, Suhas, et autres
Publié: (2023)

Auditing Prompt Caching in Language Model APIs
par: Gu, Chenchen, et autres
Publié: (2025)

Out-of-Domain Robustness via Targeted Augmentations
par: Gao, Irena, et autres
Publié: (2023)

AutoBencher: Towards Declarative Benchmark Construction
par: Li, Xiang Lisa, et autres
Publié: (2024)

Language Models with Conformal Factuality Guarantees
par: Mohri, Christopher, et autres
Publié: (2024)

Understanding Finetuning for Factual Knowledge Extraction
par: Ghosal, Gaurav, et autres
Publié: (2024)

Improving Pretraining Data Using Perplexity Correlations
par: Thrush, Tristan, et autres
Publié: (2024)

A Bitter Lesson for Data Filtering
par: Mohri, Christopher, et autres
Publié: (2026)

Stochastic Amortization: A Unified Approach to Accelerate Feature and Data Attribution
par: Covert, Ian, et autres
Publié: (2024)

Repetition Improves Language Model Embeddings
par: Springer, Jacob Mitchell, et autres
Publié: (2024)

VectorFit : Adaptive Singular & Bias Vector Fine-Tuning of Pre-trained Foundation Models
par: Hegde, Suhas G, et autres
Publié: (2025)

Provably Bounding Neural Network Preimages
par: Kotha, Suhas, et autres
Publié: (2023)

Eliciting Language Model Behaviors with Investigator Agents
par: Li, Xiang Lisa, et autres
Publié: (2025)

Scaling Laws for the Value of Individual Data Points in Machine Learning
par: Covert, Ian, et autres
Publié: (2024)

Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training
par: Liu, Hong, et autres
Publié: (2023)

Observational Scaling Laws and the Predictability of Language Model Performance
par: Ruan, Yangjun, et autres
Publié: (2024)

Scaling Self-Play with Self-Guidance
par: Bailey, Luke, et autres
Publié: (2026)

AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback
par: Dubois, Yann, et autres
Publié: (2023)

The Ideation-Execution Gap: Execution Outcomes of LLM-Generated versus Human Research Ideas
par: Si, Chenglei, et autres
Publié: (2025)

Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers
par: Si, Chenglei, et autres
Publié: (2024)

Linguistic Calibration of Long-Form Generations
par: Band, Neil, et autres
Publié: (2024)

Decoupling Exploration and Exploitation for Unsupervised Pre-training with Successor Features
par: Kim, JaeYoon, et autres
Publié: (2024)

Putting It All into Context: Simplifying Agents with LCLMs
par: Jiang, Mingjian, et autres
Publié: (2025)

Thinking Augmented Pre-training
par: Wang, Liang, et autres
Publié: (2025)

Reasoning to Learn from Latent Thoughts
par: Ruan, Yangjun, et autres
Publié: (2025)

C-LoRA: Continual Low-Rank Adaptation for Pre-trained Models
par: Zhang, Xin, et autres
Publié: (2025)

s1: Simple test-time scaling
par: Muennighoff, Niklas, et autres
Publié: (2025)

Utilizing Strategic Pre-training to Reduce Overfitting: Baguan -- A Pre-trained Weather Forecasting Model
par: Niu, Peisong, et autres
Publié: (2025)

Synthetic continued pretraining
par: Yang, Zitong, et autres
Publié: (2024)

Agentic Adversarial QA for Improving Domain-Specific LLMs
par: Grari, Vincent, et autres
Publié: (2026)

Graph-based Uncertainty Metrics for Long-form Language Model Outputs
par: Jiang, Mingjian, et autres
Publié: (2024)

Exploiting Vocabulary Frequency Imbalance in Language Model Pre-training
par: Chung, Woojin, et autres
Publié: (2025)