Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2601.17257 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909999822798848 |
|---|---|
| author | Porras-Valenzuela, Javier Hadou, Samar Ribeiro, Alejandro |
| author_facet | Porras-Valenzuela, Javier Hadou, Samar Ribeiro, Alejandro |
| contents | We introduce a constrained optimization framework for training transformers that behave like optimization descent algorithms. Specifically, we enforce layerwise descent constraints on the objective function and replace standard empirical risk minimization (ERM) with a primal-dual training scheme. This approach yields models whose intermediate representations decrease the loss monotonically in expectation across layers. We apply our method to both unrolled transformer architectures and conventional pretrained transformers on tasks of video denoising and text classification. Across these settings, we observe constrained transformers achieve stronger robustness to perturbations and maintain higher out-of-distribution generalization, while preserving in-distribution performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_17257 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | A Constrained Optimization Perspective of Unrolled Transformers Porras-Valenzuela, Javier Hadou, Samar Ribeiro, Alejandro Machine Learning We introduce a constrained optimization framework for training transformers that behave like optimization descent algorithms. Specifically, we enforce layerwise descent constraints on the objective function and replace standard empirical risk minimization (ERM) with a primal-dual training scheme. This approach yields models whose intermediate representations decrease the loss monotonically in expectation across layers. We apply our method to both unrolled transformer architectures and conventional pretrained transformers on tasks of video denoising and text classification. Across these settings, we observe constrained transformers achieve stronger robustness to perturbations and maintain higher out-of-distribution generalization, while preserving in-distribution performance. |
| title | A Constrained Optimization Perspective of Unrolled Transformers |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2601.17257 |