Image de couverture de livre

Breaking the Frozen Subspace: Importance Sampling for Low-Rank Optimization in LLM Pretraining

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Zhang, Haochen, Yin, Junze, Wang, Guanchu, Liu, Zirui, Yang, Lin F., Zhang, Tianyi, Shrivastava, Anshumali, Braverman, Vladimir
Format:	Preprint
Publié:	2025
Sujets:	Machine Learning
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

CoVE: Compressed Vocabulary Expansion Makes Better LLM-based Recommender Systems
par: Zhang, Haochen, et autres
Publié: (2025)

LeanQuant: Accurate and Scalable Large Language Model Quantization with Loss-error-aware Grid
par: Zhang, Tianyi, et autres
Publié: (2024)

Support Basis: Fast Attention Beyond Bounded Entries
par: Aliakbarpour, Maryam, et autres
Publié: (2025)

KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization
par: Zhang, Tianyi, et autres
Publié: (2024)

IDentity with Locality: An ideal hash for gene sequence search
par: Desai, Aditya, et autres
Publié: (2024)

Learning Scalable Structural Representations for Link Prediction with Bloom Signatures
par: Zhang, Tianyi, et autres
Publié: (2023)

NoMAD-Attention: Efficient LLM Inference on CPUs Through Multiply-add-free Attention
par: Zhang, Tianyi, et autres
Publié: (2024)

High-Dimensional Robust Mean Estimation with Untrusted Batches
par: Aliakbarpour, Maryam, et autres
Publié: (2026)

Sketch to Adapt: Fine-Tunable Sketches for Efficient LLM Adaptation
par: Zhang, Tianyi, et autres
Publié: (2024)

Confident or Seek Stronger: Exploring Uncertainty-Based On-device LLM Routing From Benchmarking to Generalization
par: Chuang, Yu-Neng, et autres
Publié: (2025)

70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11)
par: Zhang, Tianyi, et autres
Publié: (2025)

To Compress or Not? Pushing the Frontier of Lossless GenAI Model Weights Compression with Exponent Concentration
par: Yang, Zeyu, et autres
Publié: (2025)

Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models
par: Luo, Feng, et autres
Publié: (2026)

Efficient Alternating Minimization with Applications to Weighted Low Rank Approximation
par: Song, Zhao, et autres
Publié: (2023)

Low Rank Matrix Completion via Robust Alternating Minimization in Nearly Linear Time
par: Gu, Yuzhou, et autres
Publié: (2023)

A Dynamic Low-Rank Fast Gaussian Transform
par: Huang, Baihe, et autres
Publié: (2022)

Low-Rank Compression of Pretrained Models via Randomized Subspace Iteration
par: Pourkamali-Anaraki, Farhad
Publié: (2026)

Superintelligent Retrieval Agent: The Next Frontier of Information Retrieval
par: Yang, Zeyu, et autres
Publié: (2026)

SRLoRA: Subspace Recomposition in Low-Rank Adaptation via Importance-Based Fusion and Reinitialization
par: Yang, Haodong, et autres
Publié: (2025)

Personalizing Low-Rank Bayesian Neural Networks Via Federated Learning
par: Zhang, Boning, et autres
Publié: (2024)

Scout Before You Attend: Sketch-and-Walk Sparse Attention for Efficient LLM Inference
par: Le, Hoang Anh Duy, et autres
Publié: (2026)

DTS: Enhancing Large Reasoning Models via Decoding Tree Sketching
par: Xu, Zicheng, et autres
Publié: (2025)

GIPO: Gaussian Importance Sampling Policy Optimization
par: Lu, Chengxuan, et autres
Publié: (2026)

Stabilizing Native Low-Rank LLM Pretraining
par: Janson, Paul, et autres
Publié: (2026)

Regularizing Subspace Redundancy of Low-Rank Adaptation
par: Zhu, Yue, et autres
Publié: (2025)

From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applications
par: Jaiswal, Ajay, et autres
Publié: (2024)

Lotus: Efficient LLM Training by Randomized Low-Rank Gradient Projection with Adaptive Subspace Switching
par: Miao, Tianhao, et autres
Publié: (2026)

Assessing and Enhancing Large Language Models in Rare Disease Question-answering
par: Wang, Guanchu, et autres
Publié: (2024)

Scalable Importance Sampling in High Dimensions with Low-Rank Mixture Proposals
par: Kruse, Liam A., et autres
Publié: (2025)

CoRA: Optimizing Low-Rank Adaptation with Common Subspace of Large Language Models
par: Xiao, Xiaojun, et autres
Publié: (2024)

Mixture-of-Subspaces in Low-Rank Adaptation
par: Wu, Taiqiang, et autres
Publié: (2024)

REFRAG: Rethinking RAG based Decoding
par: Lin, Xiaoqiang, et autres
Publié: (2025)

Borrowed Geometry: Cross-Distribution Head-Importance Fingerprints of Frozen Pretrained Gemma 4 31B
par: Bektursun, Abay
Publié: (2026)

Breaking the Blocks: Continuous Low-Rank Decomposed Scaling for Unified LLM Quantization and Adaptation
par: Tang, Pingzhi, et autres
Publié: (2026)

Self-ensemble: Mitigating Confidence Mis-calibration for Large Language Models
par: Xu, Zicheng, et autres
Publié: (2025)

ESPO: Entropy Importance Sampling Policy Optimization
par: Sheng, Yuepeng, et autres
Publié: (2025)

CARAMEL: A Succinct Read-Only Lookup Table via Compressed Static Functions
par: Coleman, Benjamin, et autres
Publié: (2023)

Rethinking Importance Sampling in LLM Policy Optimization: A Cumulative Token Perspective
par: Zhang, Yuheng, et autres
Publié: (2026)

PeriodicLoRA: Breaking the Low-Rank Bottleneck in LoRA Optimization
par: Meng, Xiangdi, et autres
Publié: (2024)

Low-Rank Robust Subspace Tensor Clustering for Metro Passenger Flow Modeling
par: Hu, Jiuyun, et autres
Publié: (2024)