Imagen de Portada

Characterizing Learning Curves During Language Model Pre-Training: Learning, Forgetting, and Stability

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Chang, Tyler A., Tu, Zhuowen, Bergen, Benjamin K.
Formato:	Preprint
Publicado:	2023
Materias:	Computation and Language
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Goldfish: Monolingual Language Models for 350 Languages
por: Chang, Tyler A., et al.
Publicado: (2024)

Bigram Subnetworks: Mapping to Next Tokens in Transformer Language Models
por: Chang, Tyler A., et al.
Publicado: (2025)

A Bit of a Problem: Measurement Disparities in Dataset Sizes Across Languages
por: Arnett, Catherine, et al.
Publicado: (2024)

On the Acquisition of Shared Grammatical Representations in Bilingual Language Models
por: Arnett, Catherine, et al.
Publicado: (2025)

Exploring Forgetting in Large Language Model Pre-Training
por: Liao, Chonghua, et al.
Publicado: (2024)

Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scale
por: Michaelov, James A., et al.
Publicado: (2025)

Explaining and Mitigating Crosslingual Tokenizer Inequities
por: Arnett, Catherine, et al.
Publicado: (2025)

How Do Large Language Models Learn Concepts During Continual Pre-Training?
por: Yao, Barry Menglong, et al.
Publicado: (2026)

Do Large Language Models Exhibit Spontaneous Rational Deception?
por: Taylor, Samuel M., et al.
Publicado: (2025)

Large Language Models Pass the Turing Test
por: Jones, Cameron R., et al.
Publicado: (2025)

FOREVER: Forgetting Curve-Inspired Memory Replay for Language Model Continual Learning
por: Feng, Yujie, et al.
Publicado: (2026)

Revenge of the Fallen? Recurrent Models Match Transformers at Predicting Human Language Comprehension Metrics
por: Michaelov, James A., et al.
Publicado: (2024)

Why Does Reinforcement Learning Generalize? A Feature-Level Mechanistic Study of Post-Training in Large Language Models
por: Shi, Dan, et al.
Publicado: (2026)

Examining Forgetting in Continual Pre-training of Aligned Large Language Models
por: Li, Chen-An, et al.
Publicado: (2024)

Emergent inabilities? Inverse scaling over the course of pretraining
por: Michaelov, James A., et al.
Publicado: (2023)

Why do language models perform worse for morphologically complex languages?
por: Arnett, Catherine, et al.
Publicado: (2024)

Learning and Forgetting Unsafe Examples in Large Language Models
por: Zhao, Jiachen, et al.
Publicado: (2023)

Improving Language Models Trained on Translated Data with Continual Pre-Training and Dictionary Learning Analysis
por: Boughorbel, Sabri, et al.
Publicado: (2024)

Lies, Damned Lies, and Distributional Language Statistics: Persuasion and Deception with Large Language Models
por: Jones, Cameron R., et al.
Publicado: (2024)

Train It and Forget It: Merge Lists are Unnecessary for BPE Inference in Language Models
por: Sawada, Tomohiro, et al.
Publicado: (2025)

An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning
por: Luo, Yun, et al.
Publicado: (2023)

Synthetic Pre-Pre-Training Improves Language Model Robustness to Noisy Pre-Training Data
por: Guo, Xu, et al.
Publicado: (2026)

LoopRPT: Reinforcement Pre-Training for Looped Language Models
por: Tang, Guo, et al.
Publicado: (2026)

Offline Learning and Forgetting for Reasoning with Large Language Models
por: Ni, Tianwei, et al.
Publicado: (2025)

Learning Dynamics in Continual Pre-Training for Large Language Models
por: Wang, Xingjin, et al.
Publicado: (2025)

Pre-Trained Language Models Represent Some Geographic Populations Better Than Others
por: Dunn, Jonathan, et al.
Publicado: (2024)

Mapping Post-Training Forgetting in Language Models at Scale
por: Harmon, Jackson, et al.
Publicado: (2025)

Verifiable by Design: Aligning Language Models to Quote from Pre-Training Data
por: Zhang, Jingyu, et al.
Publicado: (2024)

Forgetting before Learning: Utilizing Parametric Arithmetic for Knowledge Updating in Large Language Models
por: Ni, Shiwen, et al.
Publicado: (2023)

EVOKE: Emotion Vocabulary Of Korean and English
por: Jung, Yoonwon, et al.
Publicado: (2026)

How Open Must Language Models be to Enable Reliable Scientific Inference?
por: Michaelov, James A., et al.
Publicado: (2026)

Forgetting Curve: A Reliable Method for Evaluating Memorization Capability for Long-context Models
por: Liu, Xinyu, et al.
Publicado: (2024)

Learning is Forgetting: LLM Training As Lossy Compression
por: Conklin, Henry C., et al.
Publicado: (2026)

Safe Reinforcement Learning with Free-form Natural Language Constraints and Pre-Trained Language Models
por: Lou, Xingzhou, et al.
Publicado: (2024)

Medical Vision-Language Pre-Training for Brain Abnormalities
por: Monajatipoor, Masoud, et al.
Publicado: (2024)

Musketeer: Joint Training for Multi-task Vision Language Model with Task Explanation Prompts
por: Zhang, Zhaoyang, et al.
Publicado: (2023)

Does GPT-4 pass the Turing test?
por: Jones, Cameron R., et al.
Publicado: (2023)

PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs
por: van der Wal, Oskar, et al.
Publicado: (2025)

Not quite Sherlock Holmes: Language model predictions do not reliably differentiate impossible from improbable events
por: Michaelov, James A., et al.
Publicado: (2025)

Circuit Stability Characterizes Language Model Generalization
por: Sun, Alan
Publicado: (2025)