Cover Image

When Does Sparsity Mitigate the Curse of Depth in LLMs

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Muhtar, Dilxat, Song, Xinyuan, Pokutta, Sebastian, Zimmer, Max, Pelleriti, Nico, Hofmann, Thomas, Liu, Shiwei
Format:	Preprint
Published:	2026
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Computational Algebra with Attention: Transformer Oracles for Border Basis Algorithms
by: Kera, Hiroshi, et al.
Published: (2025)

Why Diffusion Language Models Struggle with Truly Parallel (Non-Autoregressive) Decoding?
by: Li, Pengxiang, et al.
Published: (2026)

The Agentic Researcher: A Practical Guide to AI-Assisted Research in Mathematics and Machine Learning
by: Zimmer, Max, et al.
Published: (2026)

Approximating Latent Manifolds in Neural Networks via Vanishing Ideals
by: Pelleriti, Nico, et al.
Published: (2025)

ActTail: Global Activation Sparsity in Large Language Models
by: Hou, Wenwen, et al.
Published: (2026)

Neural Sum-of-Squares: Certifying the Nonnegativity of Polynomials with Transformers
by: Pelleriti, Nico, et al.
Published: (2025)

From Associations to Activations: Comparing Behavioral and Hidden-State Semantic Geometry in LLMs
by: Schiekiera, Louis, et al.
Published: (2026)

Diffusion Language Models Know the Answer Before Decoding
by: Li, Pengxiang, et al.
Published: (2025)

The Curse of Depth in Large Language Models
by: Sun, Wenfang, et al.
Published: (2025)

An Analysis and Mitigation of the Reversal Curse
by: Lv, Ang, et al.
Published: (2023)

Do Depth-Grown Models Overcome the Curse of Depth? An In-Depth Analysis
by: Kapl, Ferdinand, et al.
Published: (2025)

On the Byzantine-Resilience of Distillation-Based Federated Learning
by: Roux, Christophe, et al.
Published: (2024)

Benford's Curse: Tracing Digit Bias to Numerical Hallucination in LLMs
by: Shao, Jiandong, et al.
Published: (2025)

The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"
by: Berglund, Lukas, et al.
Published: (2023)

Curse of Knowledge: When Complex Evaluation Context Benefits yet Biases LLM Judges
by: Li, Weiyuan, et al.
Published: (2025)

Complementary Reinforcement Learning
by: Muhtar, Dilxat, et al.
Published: (2026)

Evaluating the Reversal Curse in Model Editing
by: Xu, Hao-Xiang, et al.
Published: (2023)

When Do LLMs Need Retrieval Augmentation? Mitigating LLMs' Overconfidence Helps Retrieval Augmentation
by: Ni, Shiyu, et al.
Published: (2024)

A Theoretical Analysis of Why Masked Diffusion Models Mitigate the Reversal Curse
by: Jeon, Moongyu, et al.
Published: (2026)

Agentic MIP Research: Accelerated Constraint Handler Generation
by: Xu, Liding, et al.
Published: (2026)

Sirius: Contextual Sparsity with Correction for Efficient LLMs
by: Zhou, Yang, et al.
Published: (2024)

When Personalization Misleads: Understanding and Mitigating Hallucinations in Personalized LLMs
by: Sun, Zhongxiang, et al.
Published: (2026)

Understanding and Mitigating Language Confusion in LLMs
by: Marchisio, Kelly, et al.
Published: (2024)

When Does Verification Pay Off? A Closer Look at LLMs as Solution Verifiers
by: Lu, Jack, et al.
Published: (2025)

The Curse of CoT: On the Limitations of Chain-of-Thought in In-Context Learning
by: Zheng, Tianshi, et al.
Published: (2025)

PERP: Rethinking the Prune-Retrain Paradigm in the Era of LLMs
by: Zimmer, Max, et al.
Published: (2023)

The Position Curse: LLMs Struggle to Locate the Last Few Items in a List
by: Zhang, Zhanqi, et al.
Published: (2026)

S-DAT: A Multilingual, GenAI-Driven Framework for Automated Divergent Thinking Assessment
by: Haase, Jennifer, et al.
Published: (2025)

MTL-LoRA: Low-Rank Adaptation for Multi-Task Learning
by: Yang, Yaming, et al.
Published: (2024)

Towards Extreme Pruning of LLMs with Plug-and-Play Mixed Sparsity
by: Xu, Chi, et al.
Published: (2025)

StreamAdapter: Efficient Test Time Adaptation from Contextual Streams
by: Muhtar, Dilxat, et al.
Published: (2024)

RECON: Robust symmetry discovery via Explicit Canonical Orientation Normalization
by: Urbano, Alonso, et al.
Published: (2025)

Mitigating Reversal Curse in Large Language Models via Semantic-aware Permutation Training
by: Guo, Qingyan, et al.
Published: (2024)

Sustainability via LLM Right-sizing
by: Haase, Jennifer, et al.
Published: (2025)

Sparse-BitNet: 1.58-bit LLMs are Naturally Friendly to Semi-Structured Sparsity
by: Zhang, Di, et al.
Published: (2026)

Multilingual Large Language Models and Curse of Multilinguality
by: Gurgurov, Daniil, et al.
Published: (2024)

LHRS-Bot: Empowering Remote Sensing with VGI-Enhanced Large Multimodal Language Model
by: Muhtar, Dilxat, et al.
Published: (2024)

Has the Creativity of Large-Language Models peaked? An analysis of inter- and intra-LLM variability
by: Haase, Jennifer, et al.
Published: (2025)

Sparse Model Soups: A Recipe for Improved Pruning via Model Averaging
by: Zimmer, Max, et al.
Published: (2023)

Compression-aware Training of Neural Networks using Frank-Wolfe
by: Zimmer, Max, et al.
Published: (2022)