Cover Image

Scaling FP8 training to trillion-token LLMs

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Fishman, Maxim, Chmiel, Brian, Banner, Ron, Soudry, Daniel
Format:	Preprint
Published:	2024
Subjects:	Machine Learning Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

FP4 All the Way: Fully Quantized Training of LLMs
by: Chmiel, Brian, et al.
Published: (2025)

Normalized Architectures are Natively 4-Bit
by: Fishman, Maxim, et al.
Published: (2026)

Minimum Variance Unbiased N:M Sparsity for the Neural Gradients
by: Chmiel, Brian, et al.
Published: (2022)

EXAQ: Exponent Aware Quantization For LLMs Acceleration
by: Shkolnik, Moran, et al.
Published: (2024)

Workspace Optimization: How to Train Your Agent
by: Sarafian, Elad, et al.
Published: (2026)

Accurate Neural Training with 4-bit Matrix Multiplications at Standard Formats
by: Chmiel, Brian, et al.
Published: (2021)

Efficient Post-training Quantization with FP8 Formats
by: Shen, Haihao, et al.
Published: (2023)

MOSS: Efficient and Accurate FP8 LLM Training with Microscaling and Automatic Scaling
by: Zhang, Yu, et al.
Published: (2025)

FP8-Flow-MoE: A Casting-Free FP8 Recipe without Double Quantization Error
by: Wang, Fengjuan, et al.
Published: (2025)

Not all tokens are needed(NAT): token efficient reinforcement learning
by: Sang, Hejian, et al.
Published: (2026)

Towards Cheaper Inference in Deep Networks with Lower Bit-Width Accumulators
by: Blumenfeld, Yaniv, et al.
Published: (2024)

COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Training
by: Xi, Haocheng, et al.
Published: (2024)

Scaling Transformer to 1M tokens and beyond with RMT
by: Bulatov, Aydar, et al.
Published: (2023)

Physics in Next-token Prediction
by: An, Hongjun, et al.
Published: (2024)

Practical FP4 Training for Large-Scale MoE Models on Hopper GPUs
by: Zhang, Wuyue, et al.
Published: (2026)

To FP8 and Back Again: Quantifying Reduced Precision Effects on LLM Training Stability
by: Lee, Joonhyung, et al.
Published: (2024)

Stable Minima Cannot Overfit in Univariate ReLU Networks: Generalization by Large Step Sizes
by: Qiao, Dan, et al.
Published: (2024)

Towards Efficient Pre-training: Exploring FP4 Precision in Large Language Models
by: Zhou, Jiecheng, et al.
Published: (2025)

Retrieval from Within: An Intrinsic Capability of Attention-Based Models
by: Hoffer, Elad, et al.
Published: (2026)

Next-token pretraining implies in-context learning
by: Riechers, Paul M., et al.
Published: (2025)

You only need 4 extra tokens: Synergistic Test-time Adaptation for LLMs
by: Xu, Yijie, et al.
Published: (2025)

From 2:4 to 8:16 sparsity patterns in LLMs for Outliers and Weights with Variance Correction
by: Maximov, Egor, et al.
Published: (2025)

The pitfalls of next-token prediction
by: Bachmann, Gregor, et al.
Published: (2024)

Looking beyond the next token
by: Thankaraj, Abitha, et al.
Published: (2025)

The Curse and Blessing of Mean Bias in FP4-Quantized LLM Training
by: Cao, Hengjie, et al.
Published: (2026)

Recipes for Pre-training LLMs with MXFP8
by: Mishra, Asit, et al.
Published: (2025)

Bio2Token: All-atom tokenization of any biomolecular structure with Mamba
by: Liu, Andrew, et al.
Published: (2024)

StreamFP: Learnable Fingerprint-guided Data Selection for Efficient Stream Learning
by: Shi, Tongjun, et al.
Published: (2024)

Pretraining large language models with MXFP4 on Native FP4 Hardware
by: Cim, Musa, et al.
Published: (2026)

The Illusion of Equivalence: Systematic FP16 Divergence in KV-Cached Autoregressive Inference
by: Chodavarapu, Ranjith, et al.
Published: (2026)

Shaping capabilities with token-level data filtering
by: Rathi, Neil, et al.
Published: (2026)

Don't flatten, tokenize! Unlocking the key to SoftMoE's efficacy in deep RL
by: Sokar, Ghada, et al.
Published: (2024)

SUN: Shared Use of Next-token Prediction for Efficient Multi-LLM Disaggregated Serving
by: Woo, Sunghyeon, et al.
Published: (2026)

FP6-LLM: Efficiently Serving Large Language Models Through FP6-Centric Algorithm-System Co-Design
by: Xia, Haojun, et al.
Published: (2024)

FP=xINT:Representing Neural Networks via Low-Bit Series Basis Functions
by: Zhang, Boyang, et al.
Published: (2024)

INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formats
by: Chen, Mengzhao, et al.
Published: (2025)

Defeating the Training-Inference Mismatch via FP16
by: Qi, Penghui, et al.
Published: (2025)

Scaling Laws for Pre-training Agents and World Models
by: Pearce, Tim, et al.
Published: (2024)

OSF: On Pre-training and Scaling of Sleep Foundation Models
by: Shuai, Zitao, et al.
Published: (2026)

AutoOR: Scalably Post-training LLMs to Autoformalize Operations Research Problems
by: Motwani, Sumeet Ramesh, et al.
Published: (2026)