Copertina

Suppressing Final Layer Hidden State Jumps in Transformer Pretraining

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Shibata, Keigo, Yano, Kazuki, Takahashi, Ryosuke, Lee, Jaesung, Ikeda, Wataru, Suzuki, Jun
Natura:	Preprint
Pubblicazione:	2026
Soggetti:	Computation and Language
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Layerwise Importance Analysis of Feed-Forward Networks in Transformer-based Language Models
di: Ikeda, Wataru, et al.
Pubblicazione: (2025)

Adapting Text LLMs to Speech via Multimodal Depth Up-Scaling
di: Yano, Kazuki, et al.
Pubblicazione: (2026)

STEP: Staged Parameter-Efficient Pre-training for Large Language Models
di: Yano, Kazuki, et al.
Pubblicazione: (2025)

TimeMachine-bench: A Benchmark for Evaluating Model Capabilities in Repository-Level Migration Tasks
di: Fujii, Ryo, et al.
Pubblicazione: (2026)

Efficient Construction of Model Family through Progressive Training Using Model Expansion
di: Yano, Kazuki, et al.
Pubblicazione: (2025)

Mining Hidden Thoughts from Texts: Evaluating Continual Pretraining with Synthetic Data for LLM Reasoning
di: Ishibashi, Yoichi, et al.
Pubblicazione: (2025)

Pre-training LLM without Learning Rate Decay Enhances Supervised Fine-Tuning
di: Yano, Kazuki, et al.
Pubblicazione: (2026)

Stabilizing Reasoning in Medical LLMs with Continued Pretraining and Reasoning Preference Optimization
di: Kawakami, Wataru, et al.
Pubblicazione: (2025)

Reconsidering Positional Supervision in Masked Diffusion Language Model Training
di: Ye, Mengyu, et al.
Pubblicazione: (2026)

An Open and Reproducible Deep Research Agent for Long-Form Question Answering
di: Yamada, Ikuya, et al.
Pubblicazione: (2025)

Transforming Hidden States into Binary Semantic Features
di: Musil, Tomáš, et al.
Pubblicazione: (2024)

Jump to Conclusions: Short-Cutting Transformers With Linear Transformations
di: Din, Alexander Yom, et al.
Pubblicazione: (2023)

States Hidden in Hidden States: LLMs Emerge Discrete State Representations Implicitly
di: Chen, Junhao, et al.
Pubblicazione: (2024)

LEAP: Layer-wise Exit-Aware Pretraining for Efficient Transformer Inference
di: Kapadia, Shashank, et al.
Pubblicazione: (2026)

LLM Hallucination Detection: A Fast Fourier Transform Method Based on Hidden Layer Temporal Signals
di: Li, Jinxin, et al.
Pubblicazione: (2025)

Mixture of Hidden-Dimensions Transformer
di: Chen, Yilong, et al.
Pubblicazione: (2024)

Curriculum-Guided Layer Scaling for Language Model Pretraining
di: Singh, Karanpartap, et al.
Pubblicazione: (2025)

The Hidden Space of Transformer Language Adapters
di: Alabi, Jesujoba O., et al.
Pubblicazione: (2024)

Hidden Heroes and Gradient Bloats: Layer-Wise Redundancy Inverts Attribution in Transformers
di: Ye, Donald
Pubblicazione: (2026)

Why Are Positional Encodings Nonessential for Deep Autoregressive Transformers? Revisiting a Petroglyph
di: Irie, Kazuki
Pubblicazione: (2024)

TopK Language Models
di: Takahashi, Ryosuke, et al.
Pubblicazione: (2025)

ELO: Efficient Layer-Specific Optimization for Continual Pretraining of Multilingual LLMs
di: Yoo, HanGyeol, et al.
Pubblicazione: (2026)

A Family of Pretrained Transformer Language Models for Russian
di: Zmitrovich, Dmitry, et al.
Pubblicazione: (2023)

Layer by Layer: Uncovering Hidden Representations in Language Models
di: Skean, Oscar, et al.
Pubblicazione: (2025)

TPTT: Transforming Pretrained Transformers into Titans
di: Furfaro, Fabien
Pubblicazione: (2025)

When Hidden States Drift: Can KV Caches Rescue Long-Range Speculative Decoding?
di: Liu, Tianyu, et al.
Pubblicazione: (2026)

Improving LLM Final Representations with Inter-Layer Geometry
di: Ulanovski, Tom, et al.
Pubblicazione: (2026)

Transformer Layers as Painters
di: Sun, Qi, et al.
Pubblicazione: (2024)

TESS 2: A Large-Scale Generalist Diffusion Language Model
di: Tae, Jaesung, et al.
Pubblicazione: (2025)

Biomed-Enriched: A Biomedical Dataset Enriched with LLMs for Pretraining and Extracting Rare and Hidden Content
di: Touchent, Rian, et al.
Pubblicazione: (2025)

Making the Most of your Model: Methods for Finetuning and Applying Pretrained Transformers
di: Yoshida, Davis
Pubblicazione: (2024)

Retrieval-Pretrained Transformer: Long-range Language Modeling with Self-retrieval
di: Rubin, Ohad, et al.
Pubblicazione: (2023)

The Curse of Popularity: Popular Entities have Catastrophic Side Effects when Deleting Knowledge from Language Models
di: Takahashi, Ryosuke, et al.
Pubblicazione: (2024)

HiddenDetect: Detecting Jailbreak Attacks against Large Vision-Language Models via Monitoring Hidden States
di: Jiang, Yilei, et al.
Pubblicazione: (2025)

Exploring Database Normalization Effects on SQL Generation
di: Kohita, Ryosuke
Pubblicazione: (2025)

Learning Less Is More: Premature Upper-Layer Attention Specialization Hurts Language Model Pretraining
di: Zhu, Jinchang, et al.
Pubblicazione: (2026)

Beyond the Final Layer: Intermediate Representations for Better Multilingual Calibration in Large Language Models
di: Zhou, Ej, et al.
Pubblicazione: (2025)

PretrainZero: Reinforcement Active Pretraining
di: Xing, Xingrun, et al.
Pubblicazione: (2025)

Where Should Diffusion Enter a Language Model? Geometry-Guided Hidden-State Replacement
di: Kong, Injin, et al.
Pubblicazione: (2026)

AERIC: Anticipatory Hidden-State Monitoring for Implicit Harmful Dialogue
di: Park, Jihyung, et al.
Pubblicazione: (2026)