Image de couverture de livre

Exploring the Benefit of Activation Sparsity in Pre-training

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Zhang, Zhengyan, Xiao, Chaojun, Qin, Qiujieli, Lin, Yankai, Zeng, Zhiyuan, Han, Xu, Liu, Zhiyuan, Xie, Ruobing, Sun, Maosong, Zhou, Jie
Format:	Preprint
Publié:	2024
Sujets:	Computation and Language Artificial Intelligence
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Variator: Accelerating Pre-trained Models with Plug-and-Play Compression Modules
par: Xiao, Chaojun, et autres
Publié: (2023)

InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory
par: Xiao, Chaojun, et autres
Publié: (2024)

BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsity
par: Song, Chenyang, et autres
Publié: (2025)

Sparsing Law: Towards Large Language Models with Greater Activation Sparsity
par: Luo, Yuqi, et autres
Publié: (2024)

Representation Learning for Natural Language Processing
par: Liu, Zhiyuan, et autres
Publié: (2020)

Robust and Scalable Model Editing for Large Language Models
par: Chen, Yingfa, et autres
Publié: (2024)

Densing Law of LLMs
par: Xiao, Chaojun, et autres
Publié: (2024)

ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity within Large Language Models
par: Song, Chenyang, et autres
Publié: (2024)

ReLU$^2$ Wins: Discovering Efficient Activation Functions for Sparse LLMs
par: Zhang, Zhengyan, et autres
Publié: (2024)

APB: Accelerating Distributed Long-Context Inference by Passing Compressed Context Blocks across GPUs
par: Huang, Yuxiang, et autres
Publié: (2025)

H-Neurons: On the Existence, Impact, and Origin of Hallucination-Associated Neurons in LLMs
par: Gao, Cheng, et autres
Publié: (2025)

Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment
par: Guo, Yiju, et autres
Publié: (2024)

Rational Decision-Making Agent with Internalized Utility Judgment
par: Ye, Yining, et autres
Publié: (2023)

Locret: Enhancing Eviction in Long-Context LLM Inference with Trained Retaining Heads on Consumer-Grade Devices
par: Huang, Yuxiang, et autres
Publié: (2024)

MAVEN-Arg: Completing the Puzzle of All-in-One Event Understanding Dataset with Event Argument Annotation
par: Wang, Xiaozhi, et autres
Publié: (2023)

Ouroboros: Generating Longer Drafts Phrase by Phrase for Faster Speculative Decoding
par: Zhao, Weilin, et autres
Publié: (2024)

StateX: Enhancing RNN Recall via Post-training State Expansion
par: Shen, Xingyu, et autres
Publié: (2025)

UltraFeedback: Boosting Language Models with Scaled AI Feedback
par: Cui, Ganqu, et autres
Publié: (2023)

CA-LoRA: Adapting Existing LoRA for Compressed LLMs to Enable Efficient Multi-Tasking on Personal Devices
par: Zhao, Weilin, et autres
Publié: (2023)

Large Language Models' Complicit Responses to Illicit Instructions across Socio-Legal Contexts
par: Wang, Xing, et autres
Publié: (2025)

KBAlign: Efficient Self Adaptation on Specific Knowledge Bases
par: Zeng, Zheni, et autres
Publié: (2024)

Investigate-Consolidate-Exploit: A General Strategy for Inter-Task Agent Self-Evolution
par: Qian, Cheng, et autres
Publié: (2024)

Exploring Forgetting in Large Language Model Pre-Training
par: Liao, Chonghua, et autres
Publié: (2024)

Mastering Text, Code and Math Simultaneously via Fusing Highly Specialized Language Models
par: Ding, Ning, et autres
Publié: (2024)

The Elephant in the Room: Rethinking the Usage of Pre-trained Language Model in Sequential Recommendation
par: Qu, Zekai, et autres
Publié: (2024)

States Hidden in Hidden States: LLMs Emerge Discrete State Representations Implicitly
par: Chen, Junhao, et autres
Publié: (2024)

DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices
par: Song, Chenyang, et autres
Publié: (2026)

Predicting Emergent Abilities with Infinite Resolution Evaluation
par: Hu, Shengding, et autres
Publié: (2023)

Tell Me More! Towards Implicit User Intention Understanding of Language Model Driven Agents
par: Qian, Cheng, et autres
Publié: (2024)

Efficient Data Learning for Open Information Extraction with Pre-trained Language Models
par: Fan, Zhiyuan, et autres
Publié: (2023)

Beyond Natural Language: LLMs Leveraging Alternative Formats for Enhanced Reasoning and Communication
par: Chen, Weize, et autres
Publié: (2024)

Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning
par: Xia, Mengzhou, et autres
Publié: (2023)

Exploring Format Consistency for Instruction Tuning
par: Liang, Shihao, et autres
Publié: (2023)

Learning to Generate Structured Output with Schema Reinforcement Learning
par: Lu, Yaxi, et autres
Publié: (2025)

KG-Infused RAG: Augmenting Corpus-Based RAG with External Knowledge Graphs
par: Wu, Dingjun, et autres
Publié: (2025)

KARE-RAG: Knowledge-Aware Refinement and Enhancement for RAG
par: Li, Yongjian, et autres
Publié: (2025)

Internet of Agents: Weaving a Web of Heterogeneous Agents for Collaborative Intelligence
par: Chen, Weize, et autres
Publié: (2024)

Ultra-FineWeb: Efficient Data Filtering and Verification for High-Quality LLM Training Data
par: Wang, Yudong, et autres
Publié: (2025)

Stuffed Mamba: Oversized States Lead to the Inability to Forget
par: Chen, Yingfa, et autres
Publié: (2024)

Advancing LLM Reasoning Generalists with Preference Trees
par: Yuan, Lifan, et autres
Publié: (2024)