Cover Image

Junk DNA Hypothesis: Pruning Small Pre-Trained Weights Irreversibly and Monotonically Impairs "Difficult" Downstream Tasks in LLMs

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Yin, Lu, Jaiswal, Ajay, Liu, Shiwei, Kundu, Souvik, Wang, Zhangyang
Format:	Preprint
Published:	2023
Subjects:	Machine Learning Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

AlphaDecay: Module-wise Weight Decay for Heavy-Tailed Balancing in LLMs
by: He, Di, et al.
Published: (2025)

Value-Based Pre-Training with Downstream Feedback
by: Ke, Shuqi, et al.
Published: (2026)

Assortment of Attention Heads: Accelerating Federated PEFT with Head Pruning and Strategic Client Selection
by: Venkatesha, Yeshwanth, et al.
Published: (2025)

From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applications
by: Jaiswal, Ajay, et al.
Published: (2024)

SEAL: Steerable Reasoning Calibration of Large Language Models for Free
by: Chen, Runjin, et al.
Published: (2025)

LLaGA: Large Language and Graph Assistant
by: Chen, Runjin, et al.
Published: (2024)

Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LN
by: Li, Pengxiang, et al.
Published: (2024)

Difficult Task Yes but Simple Task No: Unveiling the Laziness in Multimodal LLMs
by: Zhao, Sihang, et al.
Published: (2024)

SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Training
by: Huang, Tianjin, et al.
Published: (2025)

Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients
by: Zhang, Zhenyu, et al.
Published: (2024)

When Fewer Layers Break More Chains: Layer Pruning Harms Test-Time Scaling in LLMs
by: Wang, Keyu, et al.
Published: (2025)

Fast and Accurate Probing of In-Training LLMs' Downstream Performances
by: Liu, Zhichen, et al.
Published: (2026)

Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity
by: Yin, Lu, et al.
Published: (2023)

On the Hardness of Junking LLMs
by: Rando, Marco, et al.
Published: (2026)

Is C4 Dataset Optimal for Pruning? An Investigation of Calibration Data for LLM Pruning
by: Bandari, Abhinav, et al.
Published: (2024)

Pre-Trained Model Recommendation for Downstream Fine-tuning
by: Bai, Jiameng, et al.
Published: (2024)

Condense, Don't Just Prune: Enhancing Efficiency and Performance in MoE Layer Pruning
by: Cao, Mingyu, et al.
Published: (2024)

IDEA Prune: An Integrated Enlarge-and-Prune Pipeline in Generative Language Model Pretraining
by: Li, Yixiao, et al.
Published: (2025)

Understanding and Mitigating the Label Noise in Pre-training on Downstream Tasks
by: Chen, Hao, et al.
Published: (2023)

Full-Rank No More: Low-Rank Weight Training for Modern Speech Recognition Models
by: Fernandez-Lopez, Adriana, et al.
Published: (2024)

Learning to Rewrite Prompts for Bootstrapping LLMs on Downstream Tasks
by: Zhou, Qinhao, et al.
Published: (2025)

Multi-domain Knowledge Graph Collaborative Pre-training and Prompt Tuning for Diverse Downstream Tasks
by: Zhang, Yichi, et al.
Published: (2024)

CLAMP-ViT: Contrastive Data-Free Learning for Adaptive Post-Training Quantization of ViTs
by: Ramachandran, Akshat, et al.
Published: (2024)

ELAS: Efficient Pre-Training of Low-Rank Large Language Models via 2:4 Activation Sparsity
by: Li, Jiaxi, et al.
Published: (2026)

Optimising Language Models for Downstream Tasks: A Post-Training Perspective
by: Shi, Zhengyan
Published: (2025)

Thermodynamic Irreversibility of Training Algorithms
by: Ziyin, Liu, et al.
Published: (2026)

The Neural Pruning Law Hypothesis
by: Barbulescu, Eugen, et al.
Published: (2025)

Protoknowledge Shapes Behaviour of LLMs in Downstream Tasks: Memorization and Generalization with Knowledge Graphs
by: Ranaldi, Federico, et al.
Published: (2025)

VocabTailor: Dynamic Vocabulary Selection for Downstream Tasks in Small Language Models
by: Zhang, Hanling, et al.
Published: (2025)

GenBFA: An Evolutionary Optimization Approach to Bit-Flip Attacks on LLMs
by: Das, Sanjay, et al.
Published: (2024)

One LR Doesn't Fit All: Heavy-Tail Guided Layerwise Learning Rates for LLMs
by: He, Di, et al.
Published: (2026)

Downstream Transfer Attack: Adversarial Attacks on Downstream Models with Pre-trained Vision Transformers
by: Zheng, Weijie, et al.
Published: (2024)

APTBench: Benchmarking Agentic Potential of Base LLMs During Pre-Training
by: Qin, Jiarui, et al.
Published: (2025)

Effective Learning for Small Reasoning Models: An Empirical Study on 0.5B Reasoning LLMs
by: Zhuang, Xialie, et al.
Published: (2025)

Routing-Aligned Fine-Tuning for Multilingual Downstream Tasks in Mixture-of-Experts Models
by: Deng, Guanzhi, et al.
Published: (2026)

Downstream Task-Oriented Generative Model Selections on Synthetic Data Training for Fraud Detection Models
by: Cheng, Yinan, et al.
Published: (2024)

Decoupling Weighing and Selecting for Integrating Multiple Graph Pre-training Tasks
by: Fan, Tianyu, et al.
Published: (2024)

Hypothesis Testing the Circuit Hypothesis in LLMs
by: Shi, Claudia, et al.
Published: (2024)

AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models
by: Lu, Haiquan, et al.
Published: (2024)

Extracting Small Translation Specialists from LLMs by Aggressively Pruning Experts
by: Martin, Liu O., et al.
Published: (2026)