Cover Image

Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMs

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Chen, Angelica, Shwartz-Ziv, Ravid, Cho, Kyunghyun, Leavitt, Matthew L., Saphra, Naomi
Format:	Preprint
Published:	2023
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

First Tragedy, then Parse: History Repeats Itself in the New Era of Large Language Models
by: Saphra, Naomi, et al.
Published: (2023)

Dynamic Masking Rate Schedules for MLM Pretraining
by: Ankner, Zachary, et al.
Published: (2023)

Antislop: A Comprehensive Framework for Identifying and Eliminating Repetitive Patterns in Language Models
by: Paech, Samuel, et al.
Published: (2025)

Robust Bias Detection in MLMs and its Application to Human Trait Ratings
by: Shrestha, Ingroj, et al.
Published: (2025)

When Attention Collapses: How Degenerate Layers in LLMs Enable Smaller, Stronger Models
by: Sanyal, Sunny, et al.
Published: (2024)

Does Representation Matter? Exploring Intermediate Layers in Large Language Models
by: Skean, Oscar, et al.
Published: (2024)

From Tokens to Thoughts: How LLMs and Humans Trade Compression for Meaning
by: Shani, Chen, et al.
Published: (2025)

Mechanistic?
by: Saphra, Naomi, et al.
Published: (2024)

Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs
by: Nguyen, Minh Nhat, et al.
Published: (2024)

You Had One Job: Per-Task Quantization Using LLMs' Hidden Representations
by: LeVi, Amit, et al.
Published: (2025)

Latent State Models of Training Dynamics
by: Hu, Michael Y., et al.
Published: (2023)

The Illusion of Progress: Re-evaluating Hallucination Detection in LLMs
by: Janiak, Denis, et al.
Published: (2025)

ChatGPT Doesn't Trust Chargers Fans: Guardrail Sensitivity in Context
by: Li, Victoria R., et al.
Published: (2024)

Hyperparameter Loss Surfaces Are Simple Near their Optima
by: Lourie, Nicholas, et al.
Published: (2025)

Sometimes I am a Tree: Data Drives Unstable Hierarchical Generalization
by: Qin, Tian, et al.
Published: (2024)

Seq-VCR: Preventing Collapse in Intermediate Transformer Representations for Enhanced Reasoning
by: Arefin, Md Rifat, et al.
Published: (2024)

Learning to Compress: Local Rank and Information Compression in Deep Neural Networks
by: Patel, Niket, et al.
Published: (2024)

TRAM: Bridging Trust Regions and Sharpness Aware Minimization
by: Sherborne, Tom, et al.
Published: (2023)

Fast Forwarding Low-Rank Training
by: Rahamim, Adir, et al.
Published: (2024)

A superpersuasive autonomous policy debating system
by: Roush, Allen, et al.
Published: (2025)

Video Representation Learning with Joint-Embedding Predictive Architectures
by: Drozdov, Katrina, et al.
Published: (2024)

Layer by Layer: Uncovering Hidden Representations in Language Models
by: Skean, Oscar, et al.
Published: (2025)

Structural Optimization Ambiguity and Simplicity Bias in Unsupervised Neural Grammar Induction
by: Park, Jinwook, et al.
Published: (2024)

Generalization Measures for Zero-Shot Cross-Lingual Transfer
by: Bassi, Saksham, et al.
Published: (2024)

Benchmarks as Microscopes: A Call for Model Metrology
by: Saxon, Michael, et al.
Published: (2024)

The Entropy Enigma: Success and Failure of Entropy Minimization
by: Press, Ori, et al.
Published: (2024)

Two Failures of Self-Consistency in the Multi-Step Reasoning of LLMs
by: Chen, Angelica, et al.
Published: (2023)

OpenDebateEvidence: A Massive-Scale Argument Mining and Summarization Dataset
by: Roush, Allen, et al.
Published: (2024)

System-Level Natural Language Feedback
by: Yuan, Weizhe, et al.
Published: (2023)

Show Your Work with Confidence: Confidence Bands for Tuning Curves
by: Lourie, Nicholas, et al.
Published: (2023)

Efficient semantic uncertainty quantification in language models via diversity-steered sampling
by: Park, Ji Won, et al.
Published: (2025)

Latent Transfer Attack: Adversarial Examples via Generative Latent Spaces
by: Shaar, Eitan, et al.
Published: (2026)

Do Activation Verbalization Methods Convey Privileged Information?
by: Li, Millicent, et al.
Published: (2025)

Data Mixing Can Induce Phase Transitions in Knowledge Acquisition
by: Gu, Xinran, et al.
Published: (2025)

Characterizing the Predictive Impact of Modalities with Supervised Latent-Variable Modeling
by: Madaan, Divyam, et al.
Published: (2026)

Temporal Generalization: A Reality Check
by: Madaan, Divyam, et al.
Published: (2025)

Scaling Laws Are Unreliable for Downstream Tasks: A Reality Check
by: Lourie, Nicholas, et al.
Published: (2025)

On the Relationship Between the Choice of Representation and In-Context Learning
by: Marinescu, Ioana, et al.
Published: (2025)

Can Interpretation Predict Behavior on Unseen Data?
by: Li, Victoria R., et al.
Published: (2025)

Are MLMs Trapped in the Visual Room?
by: Zhang, Yazhou, et al.
Published: (2025)