Cover Image

Knowledge Entropy Decay during Language Model Pretraining Hinders New Knowledge Acquisition

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Kim, Jiyeon, Lee, Hyunji, Cho, Hyowon, Jang, Joel, Hwang, Hyeonbin, Won, Seungpil, Ahn, Youbin, Lee, Dohaeng, Seo, Minjoon
Format:	Preprint
Published:	2024
Subjects:	Computation and Language Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Latent Reasoning via Sentence Embedding Prediction
by: Hwang, Hyeonbin, et al.
Published: (2025)

Characterizing Pattern Matching and Its Limits on Compositional Task Structures
by: Chang, Hoyeon, et al.
Published: (2025)

Differential Information Distribution: A Bayesian Perspective on Direct Preference Optimization
by: Won, Yunjae, et al.
Published: (2025)

How Does Vision-Language Adaptation Impact the Safety of Vision Language Models?
by: Lee, Seongyun, et al.
Published: (2024)

KTRL+F: Knowledge-Augmented In-Document Search
by: Oh, Hanseok, et al.
Published: (2023)

Can Large Language Models Keep Up? Benchmarking Online Adaptation to Continual Knowledge Streams
by: Kim, Jiyeon, et al.
Published: (2026)

Self-Explore: Enhancing Mathematical Reasoning in Language Models with Fine-grained Rewards
by: Hwang, Hyeonbin, et al.
Published: (2024)

How Well Do Large Language Models Truly Ground?
by: Lee, Hyunji, et al.
Published: (2023)

Understanding and Enhancing Mamba-Transformer Hybrids for Memory Recall and Language Modeling
by: Lee, Hyunji, et al.
Published: (2025)

Improving Probability-based Prompt Selection Through Unified Evaluation and Analysis
by: Yang, Sohee, et al.
Published: (2023)

Exploring the Practicality of Generative Retrieval on Dynamic Corpora
by: Kim, Chaeeun, et al.
Published: (2023)

Semiparametric Token-Sequence Co-Supervision
by: Lee, Hyunji, et al.
Published: (2024)

Early Decisions Matter: Proximity Bias and Initial Trajectory Shaping in Non-Autoregressive Diffusion Language Models
by: Kim, Jiyeon, et al.
Published: (2026)

How Do Large Language Models Acquire Factual Knowledge During Pretraining?
by: Chang, Hoyeon, et al.
Published: (2024)

RouterRetriever: Routing over a Mixture of Expert Embedding Models
by: Lee, Hyunji, et al.
Published: (2024)

FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets
by: Ye, Seonghyeon, et al.
Published: (2023)

UniKnow: A Unified Framework for Reliable Language Model Behavior across Parametric and External Knowledge
by: Kim, Youna, et al.
Published: (2025)

INSTRUCTIR: A Benchmark for Instruction Following of Information Retrieval Models
by: Oh, Hanseok, et al.
Published: (2024)

Can Large Language Models Develop Gambling Addiction?
by: Lee, Seungpil, et al.
Published: (2025)

On Efficient Language and Vision Assistants for Visually-Situated Natural Language Understanding: What Matters in Reading and Reasoning
by: Kim, Geewook, et al.
Published: (2024)

Rethinking the Role of Proxy Rewards in Language Model Alignment
by: Kim, Sungdong, et al.
Published: (2024)

Reasoning Abilities of Large Language Models: In-Depth Analysis on the Abstraction and Reasoning Corpus
by: Lee, Seungpil, et al.
Published: (2024)

Toward Knowledge-Guided AI for Inverse Design in Manufacturing: A Perspective on Domain, Physics, and Human-AI Synergy
by: Lee, Hugon, et al.
Published: (2025)

Improving Calibration in Test-Time Prompt Tuning for Vision-Language Models via Data-Free Flatness-Aware Prompt Pretraining
by: Jang, Hyeonseo, et al.
Published: (2026)

DSAI: Unbiased and Interpretable Latent Feature Extraction for Data-Centric AI
by: Cho, Hyowon, et al.
Published: (2024)

The CoT Encyclopedia: Analyzing, Predicting, and Controlling how a Reasoning Model will Think
by: Lee, Seongyun, et al.
Published: (2025)

Understanding Human Perception of Music Plagiarism Through a Computational Approach
by: Hwang, Daeun, et al.
Published: (2026)

Diverse Rare Sample Generation with Pretrained GANs
by: Lee, Subeen, et al.
Published: (2024)

Knowledge Integration Decay in Search-Augmented Reasoning of Large Language Models
by: Yu, Sangwon, et al.
Published: (2026)

ARCLE: The Abstraction and Reasoning Corpus Learning Environment for Reinforcement Learning
by: Lee, Hosung, et al.
Published: (2024)

Tracing Multilingual Factual Knowledge Acquisition in Pretraining
by: Liu, Yihong, et al.
Published: (2025)

Prometheus: Inducing Fine-grained Evaluation Capability in Language Models
by: Kim, Seungone, et al.
Published: (2023)

Language Model Can Do Knowledge Tracing: Simple but Effective Method to Integrate Language Model and Knowledge Tracing Task
by: Lee, Unggi, et al.
Published: (2024)

Entropy‐Driven Crystallization of Polymeric Colloids Swollen by Photocurable Resin for Ultranarrow Stopband
by: Hyeonbin Jo, et al.
Published: (2024)

Latent Paraphrasing: Perturbation on Layers Improves Knowledge Injection in Language Models
by: Kang, Minki, et al.
Published: (2024)

Transition Metal‐Based High‐Entropy Materials for Catalysis
by: Jiwoo Lee, et al.
Published: (2025)

LangBridge: Multilingual Reasoning Without Multilingual Supervision
by: Yoon, Dongkeun, et al.
Published: (2024)

Do Modern Video-LLMs Need to Listen? A Benchmark Audit and Scalable Remedy
by: Kim, Geewook, et al.
Published: (2025)

State-Space Hierarchical Compression with Gated Attention and Learnable Sampling for Hour-Long Video Understanding in Large Multimodal Models
by: Kim, Geewook, et al.
Published: (2025)

Teaching Molecular Dynamics to a Non-Autoregressive Ionic Transport Predictor
by: Kim, Jiyeon, et al.
Published: (2026)