A Tale of Tails: Model Collapse as a Change of Scaling Laws

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Dohmatob, Elvis, Feng, Yunzhen, Yang, Pu, Charton, Francois, Kempe, Julia
Format:	Preprint
Published:	2024
Subjects:	Machine Learning Artificial Intelligence Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verification
by: Feng, Yunzhen, et al.
Published: (2024)

Model Collapse Demystified: The Case of Regression
by: Dohmatob, Elvis, et al.
Published: (2024)

Strong Model Collapse
by: Dohmatob, Elvis, et al.
Published: (2024)

Scaling Laws for Associative Memories
by: Cabannes, Vivien, et al.
Published: (2023)

Iteration Head: A Mechanistic Study of Chain-of-Thought
by: Cabannes, Vivien, et al.
Published: (2024)

Emergent properties with repeated examples
by: Charton, François, et al.
Published: (2024)

Instruction Diversity Drives Generalization To Unseen Tasks
by: Zhang, Dylan, et al.
Published: (2024)

Mission Impossible: A Statistical Perspective on Jailbreaking LLMs
by: Su, Jingtong, et al.
Published: (2024)

From Concepts to Components: Concept-Agnostic Attention Module Discovery in Transformers
by: Su, Jingtong, et al.
Published: (2025)

From Symbolic Tasks to Code Generation: Diversification Yields Better Task Performers
by: Zhang, Dylan, et al.
Published: (2024)

$\textbf{Only-IF}$:Revealing the Decisive Effect of Instruction Diversity on Generalization
by: Zhang, Dylan, et al.
Published: (2024)

Attacking Bayes: On the Adversarial Robustness of Bayesian Neural Networks
by: Feng, Yunzhen, et al.
Published: (2024)

Can Language Models Discover Scaling Laws?
by: Lin, Haowei, et al.
Published: (2025)

Efficient Refusal Ablation in LLM through Optimal Transport
by: Nanfack, Geraldin, et al.
Published: (2026)

A Hitchhiker's Guide to Scaling Law Estimation
by: Choshen, Leshem, et al.
Published: (2024)

Soft Tokens, Hard Truths
by: Butt, Natasha, et al.
Published: (2025)

Embedding Trust: Semantic Isotropy Predicts Nonfactuality in Long-Form Text Generation
by: Bhardwaj, Dhrupad, et al.
Published: (2025)

Exploring Scaling Laws for EHR Foundation Models
by: Zhang, Sheng, et al.
Published: (2025)

Theoretical Foundations of Scaling Law in Familial Models
by: Song, Huan, et al.
Published: (2025)

Unifying Learning Dynamics and Generalization in Transformers Scaling Law
by: Yang, Chiwun
Published: (2025)

P$^2$ Law: Scaling Law for Post-Training After Model Pruning
by: Chen, Xiaodong, et al.
Published: (2024)

Distillation Scaling Laws
by: Busbridge, Dan, et al.
Published: (2025)

Do Efficient Transformers Really Save Computation?
by: Yang, Kai, et al.
Published: (2024)

Scaling with Collapse: Efficient and Predictable Training of LLM Families
by: Bergsma, Shane, et al.
Published: (2025)

Learning the greatest common divisor: explaining transformer predictions
by: Charton, François
Published: (2023)

To Memorize or to Retrieve: Scaling Laws for RAG-Considerate Pretraining
by: Singh, Karan, et al.
Published: (2026)

Relative-Based Scaling Law for Neural Language Models
by: Yue, Baoqing, et al.
Published: (2025)

Observational Scaling Laws and the Predictability of Language Model Performance
by: Ruan, Yangjun, et al.
Published: (2024)

Scaling Laws for Predicting Downstream Performance in LLMs
by: Chen, Yangyi, et al.
Published: (2024)

Towards a Comprehensive Scaling Law of Mixture-of-Experts
by: Zhao, Guoliang, et al.
Published: (2025)

Batched Contextual Reinforcement: A Task-Scaling Law for Efficient Reasoning
by: Yang, Bangji, et al.
Published: (2026)

Improving the Scaling Laws of Synthetic Data with Deliberate Practice
by: Askari-Hemmat, Reyhane, et al.
Published: (2025)

What Scales in Cross-Entropy Scaling Law?
by: Yan, Junxi, et al.
Published: (2025)

Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms
by: Rafailov, Rafael, et al.
Published: (2024)

Skywork-Math: Data Scaling Laws for Mathematical Reasoning in Large Language Models -- The Story Goes On
by: Zeng, Liang, et al.
Published: (2024)

Uncovering Scaling Laws for Large Language Models via Inverse Problems
by: Verma, Arun, et al.
Published: (2025)

Provable Scaling Laws for the Test-Time Compute of Large Language Models
by: Chen, Yanxi, et al.
Published: (2024)

Scaling Law with Learning Rate Annealing
by: Tissue, Howe, et al.
Published: (2024)

Int2Int: a framework for mathematics with transformers
by: Charton, François
Published: (2025)

Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws
by: Allen-Zhu, Zeyuan, et al.
Published: (2024)