Copertina

LOST: Low-rank and Sparse Pre-training for Large Language Models

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Li, Jiaxi, Yin, Lu, Shen, Li, Xu, Jinjin, Xu, Liwu, Huang, Tianjin, Wang, Wenwu, Liu, Shiwei, Wang, Xilu
Natura:	Preprint
Pubblicazione:	2025
Soggetti:	Machine Learning
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

ELAS: Efficient Pre-Training of Low-Rank Large Language Models via 2:4 Activation Sparsity
di: Li, Jiaxi, et al.
Pubblicazione: (2026)

OWLed: Outlier-weighed Layerwise Pruning for Efficient Autonomous Driving Framework
di: Li, Jiaxi, et al.
Pubblicazione: (2024)

CLIP Brings Better Features to Visual Aesthetics Learners
di: Xu, Liwu, et al.
Pubblicazione: (2023)

Model Merging in Pre-training of Large Language Models
di: Li, Yunshui, et al.
Pubblicazione: (2025)

Infinite Sampling: Efficient and Stable Grouped RL Training for Large Language Models
di: Wang, Liangyu, et al.
Pubblicazione: (2025)

Enhancing One-shot Pruned Pre-trained Language Models through Sparse-Dense-Sparse Mechanism
di: Li, Guanchen, et al.
Pubblicazione: (2024)

Sparse is Enough in Fine-tuning Pre-trained Large Language Models
di: Song, Weixi, et al.
Pubblicazione: (2023)

MLorc: Momentum Low-rank Compression for Memory Efficient Large Language Model Adaptation
di: Shen, Wei, et al.
Pubblicazione: (2025)

Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LN
di: Li, Pengxiang, et al.
Pubblicazione: (2024)

Model Unmerging: Making Your Models Unmergeable for Secure Model Sharing
di: Wang, Zihao, et al.
Pubblicazione: (2025)

SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Training
di: Huang, Tianjin, et al.
Pubblicazione: (2025)

u-LLaVA: Unifying Multi-Modal Tasks via Large Language Model
di: Xu, Jinjin, et al.
Pubblicazione: (2023)

BiPFT: Binary Pre-trained Foundation Transformer with Low-rank Estimation of Binarization Residual Polynomials
di: Xing, Xingrun, et al.
Pubblicazione: (2023)

The Curse of Depth in Large Language Models
di: Sun, Wenfang, et al.
Pubblicazione: (2025)

Enhancing Adversarial Training via Reweighting Optimization Trajectory
di: Huang, Tianjin, et al.
Pubblicazione: (2023)

Timer: Generative Pre-trained Transformers Are Large Time Series Models
di: Liu, Yong, et al.
Pubblicazione: (2024)

Bayesian Low-rank Adaptation for Large Language Models
di: Yang, Adam X., et al.
Pubblicazione: (2023)

MIA-Tuner: Adapting Large Language Models as Pre-training Text Detector
di: Fu, Wenjie, et al.
Pubblicazione: (2024)

The Compositional Architecture of Regret in Large Language Models
di: Cui, Xiangxiang, et al.
Pubblicazione: (2025)

MIPS: a Multimodal Infinite Polymer Sequence Pre-training Framework for Polymer Property Prediction
di: Wang, Jiaxi, et al.
Pubblicazione: (2025)

DR-Encoder: Encode Low-rank Gradients with Random Prior for Large Language Models Differentially Privately
di: Wu, Huiwen, et al.
Pubblicazione: (2024)

LRTuckerRep: Low-rank Tucker Representation Model for Multi-dimensional Data Completion
di: Gong, Wenwu, et al.
Pubblicazione: (2025)

Unified Video-Language Pre-training with Synchronized Audio
di: Mo, Shentong, et al.
Pubblicazione: (2024)

PhoneLM:an Efficient and Capable Small Language Model Family through Principled Pre-training
di: Yi, Rongjie, et al.
Pubblicazione: (2024)

On the Blessing of Pre-training in Weak-to-Strong Generalization
di: Yao, Wei, et al.
Pubblicazione: (2026)

LIFT the Veil for the Truth: Principal Weights Emerge after Rank Reduction for Reasoning-Focused Supervised Fine-Tuning
di: Liu, Zihang, et al.
Pubblicazione: (2025)

Dynamic Low-Rank Sparse Adaptation for Large Language Models
di: Huang, Weizhong, et al.
Pubblicazione: (2025)

Revisiting Pre-trained Language Models for Vulnerability Detection
di: Li, Youpeng, et al.
Pubblicazione: (2025)

Towards Efficient Pre-training: Exploring FP4 Precision in Large Language Models
di: Zhou, Jiecheng, et al.
Pubblicazione: (2025)

MediSwift: Efficient Sparse Pre-trained Biomedical Language Models
di: Thangarasa, Vithursan, et al.
Pubblicazione: (2024)

Unleashing the Power of Pre-trained Language Models for Offline Reinforcement Learning
di: Shi, Ruizhe, et al.
Pubblicazione: (2023)

Machine Unlearning of Pre-trained Large Language Models
di: Yao, Jin, et al.
Pubblicazione: (2024)

LEANCODE: Understanding Models Better for Code Simplification of Pre-trained Large Language Models
di: Wang, Yan, et al.
Pubblicazione: (2025)

HLAT: High-quality Large Language Model Pre-trained on AWS Trainium
di: Fan, Haozheng, et al.
Pubblicazione: (2024)

FGBERT: Function-Driven Pre-trained Gene Language Model for Metagenomics
di: Duan, ChenRui, et al.
Pubblicazione: (2024)

The Future of Large Language Model Pre-training is Federated
di: Sani, Lorenzo, et al.
Pubblicazione: (2024)

RoseLoRA: Row and Column-wise Sparse Low-rank Adaptation of Pre-trained Language Model for Knowledge Editing and Fine-tuning
di: Wang, Haoyu, et al.
Pubblicazione: (2024)

Junk DNA Hypothesis: Pruning Small Pre-Trained Weights Irreversibly and Monotonically Impairs "Difficult" Downstream Tasks in LLMs
di: Yin, Lu, et al.
Pubblicazione: (2023)

Learning Dynamics in Continual Pre-Training for Large Language Models
di: Wang, Xingjin, et al.
Pubblicazione: (2025)

Long Chain-of-Thought Compression via Fine-Grained Group Policy Optimization
di: Han, Xinchen, et al.
Pubblicazione: (2026)