Copertina

A Study on Hidden Layer Distillation for Large Language Model Pre-Training

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Guigon, Maxime, Dixon, Lucas, Sander, Michaël E.
Natura:	Preprint
Pubblicazione:	2026
Soggetti:	Computation and Language Artificial Intelligence
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models
di: Ghandeharioun, Asma, et al.
Pubblicazione: (2024)

Pre-training Distillation for Large Language Models: A Design Space Exploration
di: Peng, Hao, et al.
Pubblicazione: (2024)

Layer by Layer: Uncovering Hidden Representations in Language Models
di: Skean, Oscar, et al.
Pubblicazione: (2025)

Legal Documents Drafting with Fine-Tuned Pre-Trained Large Language Model
di: Lin, Chun-Hsien, et al.
Pubblicazione: (2024)

Embedding-to-Prefix: Parameter-Efficient Personalization for Pre-Trained Large Language Models
di: Huber, Bernd, et al.
Pubblicazione: (2025)

Bounded Hyperbolic Tangent: A Stable and Efficient Alternative to Pre-Layer Normalization in Large Language Models
di: Byun, Hoyoon, et al.
Pubblicazione: (2025)

Evolution of Concepts in Language Model Pre-Training
di: Ge, Xuyang, et al.
Pubblicazione: (2025)

Construction of Hyper-Relational Knowledge Graphs Using Pre-Trained Large Language Models
di: Datta, Preetha, et al.
Pubblicazione: (2024)

Learning Dynamics in Continual Pre-Training for Large Language Models
di: Wang, Xingjin, et al.
Pubblicazione: (2025)

Pre-Trained Language Models for Keyphrase Prediction: A Review
di: Umair, Muhammad, et al.
Pubblicazione: (2024)

Amuro and Char: Analyzing the Relationship between Pre-Training and Fine-Tuning of Large Language Models
di: Sun, Kaiser, et al.
Pubblicazione: (2024)

Revealing the Inherent Instructability of Pre-Trained Language Models
di: An, Seokhyun, et al.
Pubblicazione: (2024)

Date Fragments: A Hidden Bottleneck of Tokenization for Temporal Reasoning
di: Bhatia, Gagan, et al.
Pubblicazione: (2025)

EasyDistill: A Comprehensive Toolkit for Effective Knowledge Distillation of Large Language Models
di: Wang, Chengyu, et al.
Pubblicazione: (2025)

Group-SAE: Efficient Training of Sparse Autoencoders for Large Language Models via Layer Groups
di: Ghilardi, Davide, et al.
Pubblicazione: (2024)

Pre-Training Curriculum for Multi-Token Prediction in Language Models
di: Aynetdinov, Ansar, et al.
Pubblicazione: (2025)

Dual-Space Knowledge Distillation for Large Language Models
di: Zhang, Songming, et al.
Pubblicazione: (2024)

MiniLLM: On-Policy Distillation of Large Language Models
di: Gu, Yuxian, et al.
Pubblicazione: (2023)

Black-Box On-Policy Distillation of Large Language Models
di: Ye, Tianzhu, et al.
Pubblicazione: (2025)

Revisiting Replay and Gradient Alignment for Continual Pre-Training of Large Language Models
di: Abbes, Istabrak, et al.
Pubblicazione: (2025)

Gecko: Versatile Text Embeddings Distilled from Large Language Models
di: Lee, Jinhyuk, et al.
Pubblicazione: (2024)

MLLM-Microscope: Unlocking Hidden Structure Within Multimodal Large Language Models
di: Mussabayev, Ravil, et al.
Pubblicazione: (2026)

Mitigating Hidden Confounding by Progressive Confounder Imputation via Large Language Models
di: Yang, Hao, et al.
Pubblicazione: (2025)

CORE: A Conceptual Reasoning Layer for Large Language Models
di: Hegde, Vishwas, et al.
Pubblicazione: (2025)

Cross-Modal Knowledge Distillation for Speech Large Language Models
di: Wang, Enzhi, et al.
Pubblicazione: (2025)

ELAD: Explanation-Guided Large Language Models Active Distillation
di: Zhang, Yifei, et al.
Pubblicazione: (2024)

Distilling Event Sequence Knowledge From Large Language Models
di: Wadhwa, Somin, et al.
Pubblicazione: (2024)

Crown, Frame, Reverse: Layer-Wise Scaling Variants for LLM Pre-Training
di: Baroian, Andrei, et al.
Pubblicazione: (2025)

PLaD: Preference-based Large Language Model Distillation with Pseudo-Preference Pairs
di: Zhang, Rongzhi, et al.
Pubblicazione: (2024)

Outlier-Safe Pre-Training for Robust 4-Bit Quantization of Large Language Models
di: Park, Jungwoo, et al.
Pubblicazione: (2025)

Pre-trained Large Language Models for Financial Sentiment Analysis
di: Luo, Wei, et al.
Pubblicazione: (2024)

Spike No More: Stabilizing the Pre-training of Large Language Models
di: Takase, Sho, et al.
Pubblicazione: (2023)

"According to ...": Prompting Language Models Improves Quoting from Pre-Training Data
di: Weller, Orion, et al.
Pubblicazione: (2023)

Spatiotemporal Hidden-State Dynamics as a Signature of Internal Reasoning in Large Language Models
di: Furuya, Kotaro, et al.
Pubblicazione: (2026)

Expediting and Elevating Large Language Model Reasoning via Hidden Chain-of-Thought Decoding
di: Liu, Tianqiao, et al.
Pubblicazione: (2024)

Dual-objective Language Models: Training Efficiency Without Overfitting
di: Samuel, David, et al.
Pubblicazione: (2025)

Unraveling Emotions with Pre-Trained Models
di: Pajón-Sanmartín, Alejandro, et al.
Pubblicazione: (2025)

BayesPrompt: Prompting Large-Scale Pre-Trained Language Models on Few-shot Inference via Debiased Domain Abstraction
di: Li, Jiangmeng, et al.
Pubblicazione: (2024)

ReAD: Reinforcement-Guided Capability Distillation for Large Language Models
di: Cheng, Xueqi, et al.
Pubblicazione: (2026)

A Multi-Layered Large Language Model Framework for Disease Prediction
di: Mohamed, Malak, et al.
Pubblicazione: (2025)