Break a Lag: Triple Exponential Moving Average for Enhanced Optimization
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2023
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866929620113162240 |
|---|---|
| author | Peleg, Roi Smadar, Yair Lazebnik, Teddy Hoogi, Assaf |
| author_facet | Peleg, Roi Smadar, Yair Lazebnik, Teddy Hoogi, Assaf |
| contents | The performance of deep learning models is critically dependent on sophisticated optimization strategies. While existing optimizers have shown promising results, many rely on first-order Exponential Moving Average (EMA) techniques, which often limit their ability to track complex gradient trends accurately. This fact can lead to a significant lag in trend identification and suboptimal optimization, particularly in highly dynamic gradient behavior. To address this fundamental limitation, we introduce Fast Adaptive Moment Estimation (FAME), a novel optimization technique that leverages the power of Triple Exponential Moving Average. By incorporating an advanced tracking mechanism, FAME enhances responsiveness to data dynamics, mitigates trend identification lag, and optimizes learning efficiency. Our comprehensive evaluation encompasses different computer vision tasks including image classification, object detection, and semantic segmentation, integrating FAME into 30 distinct architectures ranging from lightweight CNNs to Vision Transformers. Through rigorous benchmarking against state-of-the-art optimizers, FAME demonstrates superior accuracy and robustness. Notably, it offers high scalability, delivering substantial improvements across diverse model complexities, architectures, tasks, and benchmarks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2306_01423 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Break a Lag: Triple Exponential Moving Average for Enhanced Optimization Peleg, Roi Smadar, Yair Lazebnik, Teddy Hoogi, Assaf Computer Vision and Pattern Recognition Machine Learning The performance of deep learning models is critically dependent on sophisticated optimization strategies. While existing optimizers have shown promising results, many rely on first-order Exponential Moving Average (EMA) techniques, which often limit their ability to track complex gradient trends accurately. This fact can lead to a significant lag in trend identification and suboptimal optimization, particularly in highly dynamic gradient behavior. To address this fundamental limitation, we introduce Fast Adaptive Moment Estimation (FAME), a novel optimization technique that leverages the power of Triple Exponential Moving Average. By incorporating an advanced tracking mechanism, FAME enhances responsiveness to data dynamics, mitigates trend identification lag, and optimizes learning efficiency. Our comprehensive evaluation encompasses different computer vision tasks including image classification, object detection, and semantic segmentation, integrating FAME into 30 distinct architectures ranging from lightweight CNNs to Vision Transformers. Through rigorous benchmarking against state-of-the-art optimizers, FAME demonstrates superior accuracy and robustness. Notably, it offers high scalability, delivering substantial improvements across diverse model complexities, architectures, tasks, and benchmarks. |
| title | Break a Lag: Triple Exponential Moving Average for Enhanced Optimization |
| topic | Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2306.01423 |