Cover Image

Optimizing Large Language Model Training Using FP4 Quantization

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Wang, Ruizhe, Gong, Yeyun, Liu, Xiao, Zhao, Guoshuai, Yang, Ziyue, Guo, Baining, Zha, Zhengjun, Cheng, Peng
Format:	Preprint
Published:	2025
Subjects:	Machine Learning Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Beyond Sunk Costs: Boosting LLM Pre-training Efficiency via Orthogonal Growth of Mixture-of-Experts
by: Wang, Ruizhe, et al.
Published: (2025)

Improving Data and Reward Design for Scientific Reasoning in Large Language Models
by: Chen, Zijie, et al.
Published: (2026)

Ensembling Large Language Models with Process Reward-Guided Tree Search for Better Complex Reasoning
by: Park, Sungjin, et al.
Published: (2024)

Enhancing Large Language Model Performance with Gradient-Based Parameter Selection
by: Li, Haoling, et al.
Published: (2024)

SIGMA: An AI-Empowered Training Stack on Early-Life Hardware
by: Qu, Lei, et al.
Published: (2025)

Overcoming Vocabulary Mismatch: Vocabulary-agnostic Teacher Guided Language Modeling
by: Shin, Haebin, et al.
Published: (2025)

Training Matryoshka Mixture-of-Experts for Elastic Inference-Time Expert Utilization
by: Wang, Yaoxiang, et al.
Published: (2025)

MSign: An Optimizer Preventing Training Instability in Large Language Models via Stable Rank Restoration
by: Ren, Lianhai, et al.
Published: (2026)

LLM-FP4: 4-Bit Floating-Point Quantized Transformers
by: Liu, Shih-yang, et al.
Published: (2023)

APOLLO: An Optimized Training Approach for Long-form Numerical Reasoning
by: Sun, Jiashuo, et al.
Published: (2022)

Tail-Aware HiFloat4: W4A4 Post-Training Quantization for Wan2.2
by: Feng, Zhanfeng, et al.
Published: (2026)

Sequence-level Large Language Model Training with Contrastive Preference Optimization
by: Feng, Zhili, et al.
Published: (2025)

Enhancing Chain-of-Thoughts Prompting with Iterative Bootstrapping in Large Language Models
by: Sun, Jiashuo, et al.
Published: (2023)

BiasFilter: An Inference-Time Debiasing Framework for Large Language Models
by: Cheng, Xiaoqing, et al.
Published: (2025)

Sigma-MoE-Tiny Technical Report
by: Hu, Qingguo, et al.
Published: (2025)

Filling the Image Information Gap for VQA: Prompting Large Language Models to Proactively Ask Questions
by: Wang, Ziyue, et al.
Published: (2023)

Process-based Self-Rewarding Language Models
by: Zhang, Shimao, et al.
Published: (2025)

Large Language Model Bias Mitigation from the Perspective of Knowledge Editing
by: Chen, Ruizhe, et al.
Published: (2024)

CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing
by: Gou, Zhibin, et al.
Published: (2023)

EfficientQAT: Efficient Quantization-Aware Training for Large Language Models
by: Chen, Mengzhao, et al.
Published: (2024)

Velocitune: A Velocity-based Dynamic Domain Reweighting Method for Continual Pre-training
by: Luo, Zheheng, et al.
Published: (2024)

TWEO: Transformers Without Extreme Outliers Enables FP8 Training And Quantization For Dummies
by: Liang, Guang, et al.
Published: (2025)

Spatial-Temporal Large Language Model for Traffic Prediction
by: Liu, Chenxi, et al.
Published: (2024)

Black-Box Prompt Optimization: Aligning Large Language Models without Model Training
by: Cheng, Jiale, et al.
Published: (2023)

Talking to the brain: Using Large Language Models as Proxies to Model Brain Semantic Representation
by: Liu, Xin, et al.
Published: (2025)

DeepThink: Aligning Language Models with Domain-Specific User Intents
by: Li, Yang, et al.
Published: (2025)

DuQuant++: Fine-grained Rotation Enhances Microscaling FP4 Quantization
by: Lin, Haokun, et al.
Published: (2026)

Mixed-Precision Graph Neural Quantization for Low Bit Large Language Models
by: Liu, Wanlong, et al.
Published: (2025)

BiasGuard: A Reasoning-enhanced Bias Detection Tool For Large Language Models
by: Fan, Zhiting, et al.
Published: (2025)

Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph
by: Sun, Jiashuo, et al.
Published: (2023)

Sigma: Differential Rescaling of Query, Key and Value for Efficient Language Models
by: Lin, Zhenghao, et al.
Published: (2025)

Metis: Training LLMs with FP4 Quantization
by: Cao, Hengjie, et al.
Published: (2025)

DynamixSFT: Dynamic Mixture Optimization of Instruction Tuning Collections
by: Shin, Haebin, et al.
Published: (2025)

Data Mixing Agent: Learning to Re-weight Domains for Continual Pre-training
by: Yang, Kailai, et al.
Published: (2025)

EL4NER: Ensemble Learning for Named Entity Recognition via Multiple Small-Parameter Large Language Models
by: Xiao, Yuzhen, et al.
Published: (2025)

InfiR2: A Comprehensive FP8 Training Recipe for Reasoning-Enhanced Language Models
by: Wang, Wenjun, et al.
Published: (2025)

SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
by: Xiao, Guangxuan, et al.
Published: (2022)

Persona-judge: Personalized Alignment of Large Language Models via Token-level Self-judgment
by: Zhang, Xiaotian, et al.
Published: (2025)

Scaling Laws for Post Training Quantized Large Language Models
by: Xu, Zifei, et al.
Published: (2024)

Task-Stratified Knowledge Scaling Laws for Post-Training Quantized Large Language Models
by: Zhou, Chenxi, et al.
Published: (2025)