Image de couverture de livre

SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretraining

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Han, Andi, Li, Jiaxiang, Huang, Wei, Hong, Mingyi, Takeda, Akiko, Jawanpuria, Pratik, Mishra, Bamdev
Format:	Preprint
Publié:	2024
Sujets:	Machine Learning
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

A Framework for Bilevel Optimization on Riemannian Manifolds
par: Han, Andi, et autres
Publié: (2024)

Riemannian coordinate descent algorithms on matrix manifolds
par: Han, Andi, et autres
Publié: (2024)

Generalized infinite dimensional Alpha-Procrustes based geometries
par: Goomanee, Salvish, et autres
Publié: (2025)

Riemannian Optimization for Hadamard Products of Low-Rank Matrices
par: Jawanpuria, Pratik, et autres
Publié: (2026)

A Gauss-Newton Approach for Min-Max Optimization in Generative Adversarial Networks
par: Mishra, Neel, et autres
Publié: (2024)

A Riemannian Approach to Ground Metric Learning for Optimal Transport
par: Jawanpuria, Pratik, et autres
Publié: (2024)

LOFT: Low-Rank Orthogonal Fine-Tuning via Task-Aware Support Selection
par: Zhao, Lanxin, et autres
Publié: (2026)

Riemannian Federated Learning via Averaging Gradient Streams
par: Huang, Zhenwei, et autres
Publié: (2024)

Federated Learning on Riemannian Manifolds with Differential Privacy
par: Huang, Zhenwei, et autres
Publié: (2024)

Intrinsic Muon: Spectral Optimization on Riemannian Matrix Manifolds
par: Li, Yibang, et autres
Publié: (2026)

Submodular Framework for Structured-Sparse Optimal Transport
par: Manupriya, Piyushi, et autres
Publié: (2024)

Memory-Efficient LLM Pretraining via Minimalist Optimizer Design
par: Glentis, Athanasios, et autres
Publié: (2025)

UniPROT: Uniform Prototype Selection via Partial Optimal Transport with Submodular Guarantees
par: Chanda, Prateek, et autres
Publié: (2026)

Nyström Approximation on Manifolds
par: Nie, Hantao, et autres
Publié: (2026)

Efficient Optimization with Orthogonality Constraint: a Randomized Riemannian Submanifold Method
par: Han, Andi, et autres
Publié: (2025)

Scalable Parameter and Memory Efficient Pretraining for LLM: Recent Algorithmic Advances and Benchmarking
par: Glentis, Athanasios, et autres
Publié: (2025)

Modified K-means Algorithm with Local Optimality Guarantees
par: Li, Mingyi, et autres
Publié: (2025)

FedSEA: Achieving Benefit of Parallelization in Federated Online Learning
par: Sahu, Harekrushna, et autres
Publié: (2026)

MMD-Regularized Unbalanced Optimal Transport
par: Manupriya, Piyushi, et autres
Publié: (2020)

On the Role of Label Noise in the Feature Learning Process
par: Han, Andi, et autres
Publié: (2025)

AdaFish: Fast low-rank parameter-efficient fine-tuning by using second-order information
par: Hu, Jiang, et autres
Publié: (2024)

A Framework for Quantifying How Pre-Training and Context Benefit In-Context Learning
par: Song, Bingqing, et autres
Publié: (2025)

Beyond IID weights: sparse and low-rank deep Neural Networks are also Gaussian Processes
par: Nait-Saada, Thiziri, et autres
Publié: (2023)

Do pretrained Transformers Learn In-Context by Gradient Descent?
par: Shen, Lingfeng, et autres
Publié: (2023)

Adaptive regularization parameter selection for high-dimensional inverse problems: A Bayesian approach with Tucker low-rank constraints
par: Yang, Qing-Mei, et autres
Publié: (2026)

PCA recovery thresholds in low-rank matrix inference with sparse noise
par: Adomaityte, Urte, et autres
Publié: (2025)

Efficient transformer adaptation for analog in-memory computing via low-rank adapters
par: Li, Chen, et autres
Publié: (2024)

On the Feature Learning in Diffusion Models
par: Han, Andi, et autres
Publié: (2024)

Harnessing small projectors and multiple views for efficient vision pretraining
par: Agrawal, Kumar Krishna, et autres
Publié: (2023)

Universal priors: solving empirical Bayes via Bayesian inference and pretraining
par: Cannella, Nick, et autres
Publié: (2026)

Evolution Strategies for Deep RL pretraining
par: Martínez, Adrian, et autres
Publié: (2026)

Robust Least-Squares Optimization for Data-Driven Predictive Control: A Geometric Approach
par: Bharadwaj, Shreyas, et autres
Publié: (2025)

The effectiveness of MAE pre-pretraining for billion-scale pretraining
par: Singh, Mannat, et autres
Publié: (2023)

On efficiently computable functions, deep networks and sparse compositionality
par: Poggio, Tomaso
Publié: (2025)

Storing overlapping associative memories on latent manifolds in low-rank spiking networks
par: Podlaski, William F., et autres
Publié: (2024)

Physics-informed waveform inversion using pretrained wavefield neural operators
par: Huang, Xinquan, et autres
Publié: (2025)

Synthetic continued pretraining
par: Yang, Zitong, et autres
Publié: (2024)

MIRO: MultI-Reward cOnditioned pretraining improves T2I quality and efficiency
par: Dufour, Nicolas, et autres
Publié: (2025)

Convergence Error Analysis of Reflected Gradient Langevin Dynamics for Globally Optimizing Non-Convex Constrained Problems
par: Sato, Kanji, et autres
Publié: (2022)

A Tensor Residual Circuit Neural Network Factorized with Matrix Product Operation
par: Chen, Andi
Publié: (2025)