Imagen de Portada

When Attention Collapses: How Degenerate Layers in LLMs Enable Smaller, Stronger Models

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Sanyal, Sunny, Shwartz-Ziv, Ravid, Dimakis, Alexandros G., Sanghavi, Sujay
Formato:	Preprint
Publicado:	2024
Materias:	Computation and Language Artificial Intelligence Machine Learning
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Layer by Layer: Uncovering Hidden Representations in Language Models
por: Skean, Oscar, et al.
Publicado: (2025)

The Illusion of Progress: Re-evaluating Hallucination Detection in LLMs
por: Janiak, Denis, et al.
Publicado: (2025)

HiSpec: Hierarchical Speculative Decoding for LLMs
por: Kumar, Avinash, et al.
Publicado: (2025)

Upweighting Easy Samples in Fine-Tuning Mitigates Forgetting
por: Sanyal, Sunny, et al.
Publicado: (2025)

From Tokens to Thoughts: How LLMs and Humans Trade Compression for Meaning
por: Shani, Chen, et al.
Publicado: (2025)

How to Train Your Advisor: Steering Black-Box LLMs with Advisor Models
por: Asawa, Parth, et al.
Publicado: (2025)

In-Context Learning with Transformers: Softmax Attention Adapts to Function Lipschitzness
por: Collins, Liam, et al.
Publicado: (2024)

Entropy Aware Reward Guidance for Diffusion Language Model Alignment
por: Tejaswi, Atula, et al.
Publicado: (2026)

Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coin
por: Queipo-de-Llano, Enrique, et al.
Publicado: (2025)

SVFT: Parameter-Efficient Fine-Tuning with Singular Vectors
por: Lingam, Vijay, et al.
Publicado: (2024)

Layer Importance for Mathematical Reasoning is Forged in Pre-Training and Invariant after Post-Training
por: Nepal, Aadim, et al.
Publicado: (2025)

Does Representation Matter? Exploring Intermediate Layers in Large Language Models
por: Skean, Oscar, et al.
Publicado: (2024)

OpenDebateEvidence: A Massive-Scale Argument Mining and Summarization Dataset
por: Roush, Allen, et al.
Publicado: (2024)

Enabling Approximate Joint Sampling in Diffusion LMs
por: Bansal, Parikshit, et al.
Publicado: (2025)

Blar-SQL: Faster, Stronger, Smaller NL2SQL
por: Domínguez, José Manuel, et al.
Publicado: (2024)

Reverse Thinking Makes LLMs Stronger Reasoners
por: Chen, Justin Chih-Yao, et al.
Publicado: (2024)

Antislop: A Comprehensive Framework for Identifying and Eliminating Repetitive Patterns in Language Models
por: Paech, Samuel, et al.
Publicado: (2025)

Hidden in the Haystack: Smaller Needles are More Difficult for LLMs to Find
por: Bianchi, Owen, et al.
Publicado: (2025)

Geometric Median Matching for Robust k-Subset Selection from Noisy Data
por: Acharya, Anish, et al.
Publicado: (2025)

Geometric Median (GM) Matching for Robust Data Pruning
por: Acharya, Anish, et al.
Publicado: (2024)

When Weak LLMs Speak with Confidence, Preference Alignment Gets Stronger
por: Afzali, Amirabbas, et al.
Publicado: (2026)

JEPA as a Neural Tokenizer: Learning Robust Speech Representations with Density Adaptive Attention
por: Ioannides, Georgios, et al.
Publicado: (2025)

When Correct Beliefs Collapse: Epistemic Resilience of LLMs under Clinical Pressure
por: Xiao, Boyu, et al.
Publicado: (2026)

Variance-Covariance Regularization Improves Representation Learning
por: Zhu, Jiachen, et al.
Publicado: (2023)

Enhancing Generalization in Chain of Thought Reasoning for Smaller Models
por: Yin, Maxwell J., et al.
Publicado: (2025)

How to Synthesize Text Data without Model Collapse?
por: Zhu, Xuekai, et al.
Publicado: (2024)

RARe: Retrieval Augmented Retrieval with In-Context Examples
por: Tejaswi, Atula, et al.
Publicado: (2024)

Beyond Answers: Transferring Reasoning Capabilities to Smaller LLMs Using Multi-Teacher Knowledge Distillation
por: Tian, Yijun, et al.
Publicado: (2024)

Seq-VCR: Preventing Collapse in Intermediate Transformer Representations for Enhanced Reasoning
por: Arefin, Md Rifat, et al.
Publicado: (2024)

Video Representation Learning with Joint-Embedding Predictive Architectures
por: Drozdov, Katrina, et al.
Publicado: (2024)

The Price of Format: Diversity Collapse in LLMs
por: Yun, Longfei, et al.
Publicado: (2025)

Integrating Arithmetic Learning Improves Mathematical Reasoning in Smaller Models
por: Gangwar, Neeraj, et al.
Publicado: (2025)

Debate Helps Weak Judges Reward Stronger Models
por: Elasky, Ethan, et al.
Publicado: (2026)

Calibration Collapse Under Sycophancy Fine-Tuning: How Reward Hacking Breaks Uncertainty Quantification in LLMs
por: Sahoo, Subramanyam
Publicado: (2026)

AI Must Embrace Specialization via Superhuman Adaptable Intelligence
por: Goldfeder, Judah, et al.
Publicado: (2026)

S2WTM: Spherical Sliced-Wasserstein Autoencoder for Topic Modeling
por: Adhya, Suman, et al.
Publicado: (2025)

Beyond the Loss Curve: Scaling Laws, Active Learning, and the Limits of Learning from Exact Posteriors
por: Khorasani, Arian, et al.
Publicado: (2026)

When Bias Pretends to Be Truth: How Spurious Correlations Undermine Hallucination Detection in LLMs
por: Wang, Shaowen, et al.
Publicado: (2025)

Consistent Diffusion Meets Tweedie: Training Exact Ambient Diffusion Models with Noisy Data
por: Daras, Giannis, et al.
Publicado: (2024)

When Attention Sink Emerges in Language Models: An Empirical View
por: Gu, Xiangming, et al.
Publicado: (2024)