Cover Image

Compressing Large Language Models using Low Rank and Low Precision Decomposition

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Saha, Rajarshi, Sagan, Naomi, Srivastava, Varun, Goldsmith, Andrea J., Pilanci, Mert
Format:	Preprint
Published:	2024
Subjects:	Machine Learning Artificial Intelligence Optimization and Control
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

From Complexity to Clarity: Analytical Expressions of Deep Neural Network Weights via Clifford's Geometric Algebra and Convexity
by: Pilanci, Mert
Published: (2023)

A Library of Mirrors: Deep Neural Nets in Low Dimensions are Convex Lasso Models with Reflection Features
by: Zeger, Emi, et al.
Published: (2024)

Optimizer-Induced Mode Connectivity: From AdamW to Muon
by: Zhang, Fangzhao, et al.
Published: (2026)

Unbiased Gradient Low-Rank Projection
by: Pan, Rui, et al.
Published: (2025)

Analyzing Neural Network-Based Generative Diffusion Models through Convex Optimization
by: Zhang, Fangzhao, et al.
Published: (2024)

Q3R: Quadratic Reweighted Rank Regularizer for Effective Low-Rank Training
by: Ghosh, Ipsita, et al.
Published: (2025)

Federated Dynamical Low-Rank Training with Global Loss Convergence Guarantees
by: Schotthöfer, Steffen, et al.
Published: (2024)

Convex Relaxations of ReLU Neural Networks Approximate Global Optima in Polynomial Time
by: Kim, Sungyoon, et al.
Published: (2024)

Riemannian Preconditioned LoRA for Fine-Tuning Foundation Models
by: Zhang, Fangzhao, et al.
Published: (2024)

A Recovery Guarantee for Sparse Neural Networks
by: Fridovich-Keil, Sara, et al.
Published: (2025)

PoLAR: Polar-Decomposed Low-Rank Adapter Representation
by: Lion, Kai, et al.
Published: (2025)

Faster Convergence of Stochastic Accelerated Gradient Descent under Interpolation
by: Mishkin, Aaron, et al.
Published: (2024)

Active Learning of Deep Neural Networks via Gradient-Free Cutting Planes
by: Zhang, Erica, et al.
Published: (2024)

CRONOS: Enhancing Deep Learning with Scalable GPU Accelerated Convex Neural Networks
by: Feng, Miria, et al.
Published: (2024)

Adversarial Training of Two-Layer Polynomial and ReLU Activation Networks via Convex Optimization
by: Kuelbs, Daniel, et al.
Published: (2024)

Weighted Low-rank Approximation via Stochastic Gradient Descent on Manifolds
by: Xu, Conglong, et al.
Published: (2025)

Learning to Deliver: a Foundation Model for the Montreal Capacitated Vehicle Routing Problem
by: Chin, Samuel J. K., et al.
Published: (2024)

GANQ: GPU-Adaptive Non-Uniform Quantization for Large Language Models
by: Zhao, Pengxiang, et al.
Published: (2025)

GNMR: Runtime Stability Control for Low-Precision Large Language Model Training
by: Kong, Boao, et al.
Published: (2026)

Uncovering Symmetry Transfer in Large Language Models via Layer-Peeled Optimization
by: Du, Zhehang, et al.
Published: (2026)

Robust Second-Order Nonconvex Optimization and Its Application to Low Rank Matrix Sensing
by: Li, Shuyao, et al.
Published: (2024)

From Large Language Models and Optimization to Decision Optimization CoPilot: A Research Manifesto
by: Wasserkrug, Segev, et al.
Published: (2024)

AdLoCo: adaptive batching significantly improves communications efficiency and convergence for Large Language Models
by: Kutuzov, Nikolay, et al.
Published: (2025)

StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Models
by: Yu, Dingzhi, et al.
Published: (2026)

Greedy Low-Rank Gradient Compression for Distributed Learning with Convergence Guarantees
by: Chen, Chuyan, et al.
Published: (2025)

From Automation to Autonomy in Smart Manufacturing: A Bayesian Optimization Framework for Modeling Multi-Objective Experimentation and Sequential Decision Making
by: Asru, Avijit Saha, et al.
Published: (2025)

Co-Learning Port-Hamiltonian Systems and Optimal Energy-Shaping Control
by: Kamboj, Ankur, et al.
Published: (2026)

Towards Faster Decentralized Stochastic Optimization with Communication Compression
by: Islamov, Rustem, et al.
Published: (2024)

One Rank at a Time: Cascading Error Dynamics in Sequential Learning
by: Vandchali, Mahtab Alizadeh, et al.
Published: (2025)

Randomized Geometric Algebra Methods for Convex Neural Networks
by: Wang, Yifei, et al.
Published: (2024)

Capabilities of Large Language Models in Control Engineering: A Benchmark Study on GPT-4, Claude 3 Opus, and Gemini 1.0 Ultra
by: Kevian, Darioush, et al.
Published: (2024)

Data-Driven Portfolio Management for Motion Pictures Industry: A New Data-Driven Optimization Methodology Using a Large Language Model as the Expert
by: Alipour-Vaezi, Mohammad, et al.
Published: (2024)

ARO: A New Lens On Matrix Optimization For Large Models
by: Gong, Wenbo, et al.
Published: (2026)

Low-Rank Extragradient Method for Nonsmooth and Low-Rank Matrix Optimization Problems
by: Garber, Dan, et al.
Published: (2022)

Low-Rank Mirror-Prox for Nonsmooth and Low-Rank Matrix Optimization Problems
by: Garber, Dan, et al.
Published: (2022)

MiMuon: Mixed Muon Optimizer with Improved Generalization for Large Models
by: Huang, Feihu, et al.
Published: (2026)

Reward Collapse in Aligning Large Language Models
by: Song, Ziang, et al.
Published: (2023)

To Cool or not to Cool? Temperature Network Meets Large Foundation Models via DRO
by: Qiu, Zi-Hao, et al.
Published: (2024)

Temporal Difference Learning with Compressed Updates: Error-Feedback meets Reinforcement Learning
by: Mitra, Aritra, et al.
Published: (2023)

The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Training
by: Wang, Jinbo, et al.
Published: (2025)