Imagen de Portada

Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Liu, Hong, Li, Zhiyuan, Hall, David, Liang, Percy, Ma, Tengyu
Formato:	Preprint
Publicado:	2023
Materias:	Machine Learning Computation and Language Optimization and Control
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Scalable Second-order Riemannian Optimization for $K$-means Clustering
por: Xu, Peng, et al.
Publicado: (2025)

humancompatible.train: Implementing Optimization Algorithms for Stochastically-Constrained Stochastic Optimization Problems
por: Kliachkin, Andrii, et al.
Publicado: (2025)

Universal Online Convex Optimization Meets Second-order Bounds
por: Zhang, Lijun, et al.
Publicado: (2021)

Adaptive and Optimal Second-order Optimistic Methods for Minimax Optimization
por: Jiang, Ruichen, et al.
Publicado: (2024)

PROMISE: Preconditioned Stochastic Optimization Methods by Incorporating Scalable Curvature Estimates
por: Frangella, Zachary, et al.
Publicado: (2023)

On Convergence of Adam for Stochastic Optimization under Relaxed Assumptions
por: Hong, Yusu, et al.
Publicado: (2024)

Second-order Conditional Gradient Sliding
por: Carderera, Alejandro, et al.
Publicado: (2020)

Second-order Optimization under Heavy-Tailed Noise: Hessian Clipping and Sample Complexity Limits
por: Sadiev, Abdurakhmon, et al.
Publicado: (2025)

Distributional Surgery for Language Model Activations
por: Nguyen, Bao, et al.
Publicado: (2025)

Bregman Linearized Augmented Lagrangian Method for Nonconvex Constrained Stochastic Zeroth-order Optimization
por: Shi, Qiankun, et al.
Publicado: (2025)

Variance Reduction and Low Sample Complexity in Stochastic Optimization via Proximal Point Method
por: Liang, Jiaming
Publicado: (2024)

Solving General Natural-Language-Description Optimization Problems with Large Language Models
por: Zhang, Jihai, et al.
Publicado: (2024)

On the Stochastic (Variance-Reduced) Proximal Gradient Method for Regularized Expected Reward Optimization
por: Liang, Ling, et al.
Publicado: (2024)

Demystifying Manifold Constraints in LLM Pre-training
por: An, Kang, et al.
Publicado: (2026)

Scalable First-order Method for Certifying Optimal k-Sparse GLMs
por: Liu, Jiachang, et al.
Publicado: (2025)

Improved Learning Rates for Stochastic Optimization
por: Li, Shaojie, et al.
Publicado: (2021)

Wasserstein Distributionally Robust Regret Optimization for Reinforcement Learning from Human Feedback
por: Wang, Yikai, et al.
Publicado: (2026)

A Unified Understanding of Offline Data Selection and Online Self-refining Generation for Post-training LLMs
por: Xiao, Quan, et al.
Publicado: (2025)

Stochastic-Constrained Stochastic Optimization with Markovian Data
por: Kim, Yeongjong, et al.
Publicado: (2023)

Stochastic Zeroth order Descent with Structured Directions
por: Rando, Marco, et al.
Publicado: (2022)

Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective
por: Wen, Kaiyue, et al.
Publicado: (2024)

A Retrospective Approximation Approach for Smooth Stochastic Optimization
por: Newton, David, et al.
Publicado: (2021)

Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models
por: Kunstner, Frederik, et al.
Publicado: (2024)

First and Second Order Approximations to Stochastic Gradient Descent Methods with Momentum Terms
por: Lu, Eric
Publicado: (2025)

A First-order Generative Bilevel Optimization Framework for Diffusion Models
por: Xiao, Quan, et al.
Publicado: (2025)

Scalable Kernel Inverse Optimization
por: Long, Youyuan, et al.
Publicado: (2024)

Oblivious Stochastic Composite Optimization
por: Lezane, Clément, et al.
Publicado: (2023)

Tuning-Free Stochastic Optimization
por: Khaled, Ahmed, et al.
Publicado: (2024)

Stochastic Optimization with Random Search
por: Chayti, El Mahdi, et al.
Publicado: (2025)

Adaptive Algorithms with Sharp Convergence Rates for Stochastic Hierarchical Optimization
por: Gong, Xiaochuan, et al.
Publicado: (2025)

An Accelerated Algorithm for Stochastic Bilevel Optimization under Unbounded Smoothness
por: Gong, Xiaochuan, et al.
Publicado: (2024)

Implicit Bias of AdamW: $\ell_\infty$ Norm Constrained Optimization
por: Xie, Shuo, et al.
Publicado: (2024)

OptiChat: Bridging Optimization Models and Practitioners with Large Language Models
por: Chen, Hao, et al.
Publicado: (2025)

Sign-Based Optimizers Are Effective Under Heavy-Tailed Noise
por: Yu, Dingzhi, et al.
Publicado: (2026)

A Split-Client Approach to Second-Order Optimization
por: Chayti, El Mahdi, et al.
Publicado: (2025)

Near-Optimal Decentralized Stochastic Nonconvex Optimization with Heavy-Tailed Noise
por: Wang, Menglian, et al.
Publicado: (2026)

A Hessian-Aware Stochastic Differential Equation for Modelling SGD
por: Li, Xiang, et al.
Publicado: (2024)

Stochastic Difference-of-Convex Optimization with Momentum
por: Chayti, El Mahdi, et al.
Publicado: (2025)

The Price of Adaptivity in Stochastic Convex Optimization
por: Carmon, Yair, et al.
Publicado: (2024)

Accelerated Parameter-Free Stochastic Optimization
por: Kreisler, Itai, et al.
Publicado: (2024)