Cover Image

Momentum Streams for Optimizer-Inspired Transformers

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Gai, Jingchu, Huang, Nai-Chieh, Wu, Jiayun
Format:	Preprint
Published:	2026
Subjects:	Machine Learning Artificial Intelligence Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

R$^2$PO: Decoupling Training Trajectories from Inference Responses for LLM Reasoning
by: Wang, Jingchu, et al.
Published: (2026)

Residual Stream Duality in Modern Transformer Architectures
by: Zhang, Yifan
Published: (2026)

Specialising and Analysing Instruction-Tuned and Byte-Level Language Models for Organic Reaction Prediction
by: Pang, Jiayun, et al.
Published: (2024)

Taming Momentum: Rethinking Optimizer States Through Low-Rank Approximation
by: Wang, Zhengbo, et al.
Published: (2026)

The Dual-Stream Transformer: Channelized Architecture for Interpretable Language Modeling
by: Kerce, J. Clayton, et al.
Published: (2026)

MetaGreen: Meta-Learning Inspired Transformer Selection for Green Semantic Communication
by: Mukherjee, Shubhabrata, et al.
Published: (2024)

SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Training
by: Huang, Tianjin, et al.
Published: (2025)

State Stream Transformer (SST) : Emergent Metacognitive Behaviours Through Latent State Persistence
by: Aviss, Thea
Published: (2025)

Modular Multi-Task Learning for Chemical Reaction Prediction
by: Pang, Jiayun, et al.
Published: (2026)

A Framework for Inference Inspired by Human Memory Mechanisms
by: Zeng, Xiangyu, et al.
Published: (2023)

Fundamental Limits of Prompt Tuning Transformers: Universality, Capacity and Efficiency
by: Hu, Jerry Yao-Chieh, et al.
Published: (2024)

Klear-Reasoner: Advancing Reasoning Capability via Gradient-Preserving Clipping Policy Optimization
by: Su, Zhenpeng, et al.
Published: (2025)

CLIPO: Contrastive Learning in Policy Optimization Generalizes RLVR
by: Cui, Sijia, et al.
Published: (2026)

Momentum Point-Perplexity Mechanics in Large Language Models
by: Tomaz, Lorenzo, et al.
Published: (2025)

Strategic Fusion Optimizes Transformer Compression
by: Rahman, Md Shoaibur
Published: (2025)

Zero-Training Temporal Drift Detection for Transformer Sentiment Models: A Comprehensive Analysis on Authentic Social Media Streams
by: Bansal, Aayam, et al.
Published: (2025)

Evolving Demonstration Optimization for Chain-of-Thought Feature Transformation
by: Wang, Xinyuan, et al.
Published: (2026)

PRISM: A Geometric Risk Bound that Decomposes Drift into Scale, Shape, and Head
by: Lin, Chieh-Yen, et al.
Published: (2026)

FOREVER: Forgetting Curve-Inspired Memory Replay for Language Model Continual Learning
by: Feng, Yujie, et al.
Published: (2026)

Grouped Sequency-arranged Rotation: Optimizing Rotation Transformation for Quantization for Free
by: Choi, Euntae, et al.
Published: (2025)

Quantum-Inspired Differentiable Integral Neural Networks (QIDINNs): A Feynman-Based Architecture for Continuous Learning Over Streaming Data
by: Dapena, Oscar Boullosa
Published: (2025)

Stream of Search (SoS): Learning to Search in Language
by: Gandhi, Kanishk, et al.
Published: (2024)

MomentumSMoE: Integrating Momentum into Sparse Mixture of Experts
by: Teo, Rachel S. Y., et al.
Published: (2024)

SwiftKV: Fast Prefill-Optimized Inference with Knowledge-Preserving Model Transformation
by: Qiao, Aurick, et al.
Published: (2024)

KnowHalu: Hallucination Detection via Multi-Form Knowledge Based Factual Checking
by: Zhang, Jiawei, et al.
Published: (2024)

LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization
by: Yen, Jui-Nan, et al.
Published: (2024)

Linear Chain Transformation: Expanding Optimization Dynamics for Fine-Tuning Large Language Models
by: Wang, Yulong, et al.
Published: (2024)

Human Inspired Progressive Alignment and Comparative Learning for Grounded Word Acquisition
by: Bao, Yuwei, et al.
Published: (2023)

Sketch-of-Thought: Efficient LLM Reasoning with Adaptive Cognitive-Inspired Sketching
by: Aytes, Simon A., et al.
Published: (2025)

Intrinsic Self-Correction in LLMs: Towards Explainable Prompting via Mechanistic Interpretability
by: Lee, Yu-Ting, et al.
Published: (2025)

Speculative Streaming: Fast LLM Inference without Auxiliary Models
by: Bhendawade, Nikhil, et al.
Published: (2024)

A Meta-Learning Perspective on Transformers for Causal Language Modeling
by: Wu, Xinbo, et al.
Published: (2023)

Discovering Interpretable Algorithms by Decompiling Transformers to RASP
by: Huang, Xinting, et al.
Published: (2026)

How Do Transformers Learn Variable Binding in Symbolic Programs?
by: Wu, Yiwei, et al.
Published: (2025)

ROM: Real-time Overthinking Mitigation via Streaming Detection and Intervention
by: Wang, Xinyan, et al.
Published: (2026)

Stream Aligner: Efficient Sentence-Level Alignment via Distribution Induction
by: Lou, Hantao, et al.
Published: (2025)

DCRM: A Heuristic to Measure Response Pair Quality in Preference Optimization
by: Huang, Chengyu, et al.
Published: (2025)

Adaptive Social Learning via Mode Policy Optimization for Language Agents
by: Wang, Minzheng, et al.
Published: (2025)

AMUSED: A Multi-Stream Vector Representation Method for Use in Natural Dialogue
by: Kumar, Gaurav, et al.
Published: (2019)

Transformers Learn Low Sensitivity Functions: Investigations and Implications
by: Vasudeva, Bhavya, et al.
Published: (2024)