Image de couverture de livre

TEON: Tensorized Orthonormalization Beyond Layer-Wise Muon for Large Language Model Pre-Training

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Zhang, Ruijie, Zhao, Yequan, Liu, Ziyue, Wang, Zhengyang, Li, Dongyang, Su, Yupeng, Liu, Sijia, Zhang, Zheng
Format:	Preprint
Publié:	2026
Sujets:	Machine Learning Artificial Intelligence
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Muon$^2$: Boosting Muon via Adaptive Second-Moment Preconditioning
par: Liu, Ziyue, et autres
Publié: (2026)

MuonQ: Enhancing Low-Bit Muon Quantization via Directional Fidelity Optimization
par: Su, Yupeng, et autres
Publié: (2026)

MUON+: Towards More Effective Muon via One Additional Normalization Step for LLM Pre-training
par: Zhang, Ruijie, et autres
Publié: (2026)

LaX: Boosting Low-Rank Training of Foundation Models via Latent Crossing
par: Zhang, Ruijie, et autres
Publié: (2025)

CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activation
par: Liu, Ziyue, et autres
Publié: (2025)

ReCoVer: Resilient LLM Pre-Training System via Fault-Tolerant Collective and Versatile Workload
par: Liu, Ziyue, et autres
Publié: (2026)

Tensor-Compressed and Fully-Quantized Training of Neural PDE Solvers
par: Lu, Jinming, et autres
Publié: (2025)

QuZO: Quantized Zeroth-Order Fine-Tuning for Large Language Models
par: Zhou, Jiajun, et autres
Publié: (2025)

BOOST: BOttleneck-Optimized Scalable Training Framework for Low-Rank Large Language Models
par: Wang, Zhengyang, et autres
Publié: (2025)

Hephaestus: Improving Fundamental Agent Capabilities of Large Language Models through Continual Pre-Training
par: Zhuang, Yuchen, et autres
Publié: (2025)

Large EEG-U-Transformer for Time-Step Level Detection Without Pre-Training
par: Wu, Kerui, et autres
Publié: (2025)

Crown, Frame, Reverse: Layer-Wise Scaling Variants for LLM Pre-Training
par: Baroian, Andrei, et autres
Publié: (2025)

Muon is Scalable for LLM Training
par: Liu, Jingyuan, et autres
Publié: (2025)

Talking to the brain: Using Large Language Models as Proxies to Model Brain Semantic Representation
par: Liu, Xin, et autres
Publié: (2025)

RankGuide: Tensor-Rank-Guided Routing and Steering for Efficient Reasoning
par: Tian, Jiayi, et autres
Publié: (2026)

CoMERA: Computing- and Memory-Efficient Training via Rank-Adaptive Tensor Optimization
par: Yang, Zi, et autres
Publié: (2024)

A Study on Hidden Layer Distillation for Large Language Model Pre-Training
par: Guigon, Maxime, et autres
Publié: (2026)

PTQTP: Post-Training Quantization to Trit-Planes for Large Language Models
par: Xiao, He, et autres
Publié: (2025)

Real-Time FJ/MAC PDE Solvers via Tensorized, Back-Propagation-Free Optical PINN Training
par: Zhao, Yequan, et autres
Publié: (2023)

Scalable Back-Propagation-Free Training of Optical Physics-Informed Neural Networks
par: Zhao, Yequan, et autres
Publié: (2025)

SkipKV: Selective Skipping of KV Generation and Storage for Efficient Inference with Large Reasoning Models
par: Tian, Jiayi, et autres
Publié: (2025)

Improve Decoding Factuality by Token-wise Cross Layer Entropy of Large Language Models
par: Wu, Jialiang, et autres
Publié: (2025)

Layer-Order Inversion: Rethinking Latent Multi-Hop Reasoning in Large Language Models
par: Liu, Xukai, et autres
Publié: (2026)

Dion: Distributed Orthonormalized Updates
par: Ahn, Kwangjun, et autres
Publié: (2025)

APTQ: Attention-aware Post-Training Mixed-Precision Quantization for Large Language Models
par: Guan, Ziyi, et autres
Publié: (2024)

Full-ECE: A Metric For Token-level Calibration on Large Language Models
par: Liu, Han, et autres
Publié: (2024)

Pre-Trained Vision-Language Models as Partial Annotators
par: Wang, Qian-Wei, et autres
Publié: (2024)

LoRETTA: Low-Rank Economic Tensor-Train Adaptation for Ultra-Low-Parameter Fine-Tuning of Large Language Models
par: Yang, Yifan, et autres
Publié: (2024)

Large Language Model as a Policy Teacher for Training Reinforcement Learning Agents
par: Zhou, Zihao, et autres
Publié: (2023)

AdaZeta: Adaptive Zeroth-Order Tensor-Train Adaption for Memory-Efficient Large Language Models Fine-Tuning
par: Yang, Yifan, et autres
Publié: (2024)

Modality Gap-Driven Subspace Alignment Training Paradigm For Multimodal Large Language Models
par: Yu, Xiaomin, et autres
Publié: (2026)

Unleashing The Power of Pre-Trained Language Models for Irregularly Sampled Time Series
par: Zhang, Weijia, et autres
Publié: (2024)

The Quantum Sieve Tracer: A Hybrid Framework for Layer-Wise Activation Tracing in Large Language Models
par: Pan, Jonathan
Publié: (2026)

WilKE: Wise-Layer Knowledge Editor for Lifelong Knowledge Editing
par: Hu, Chenhui, et autres
Publié: (2024)

Beyond Scaling: Measuring and Predicting the Upper Bound of Knowledge Retention in Language Model Pre-Training
par: Jiang, Changhao, et autres
Publié: (2025)

How to inject knowledge efficiently? Knowledge Infusion Scaling Law for Pre-training Large Language Models
par: Lv, Kangtao, et autres
Publié: (2025)

Quantization-Aware and Tensor-Compressed Training of Transformers for Natural Language Understanding
par: Yang, Zi, et autres
Publié: (2023)

Large Language Model with Region-guided Referring and Grounding for CT Report Generation
par: Chen, Zhixuan, et autres
Publié: (2024)

Benchmarking Benchmark Leakage in Large Language Models
par: Xu, Ruijie, et autres
Publié: (2024)

Advancing the Robustness of Large Language Models through Self-Denoised Smoothing
par: Ji, Jiabao, et autres
Publié: (2024)