Fast and Stable Gradient Approximation for Bilinear Forms of Hermitian Matrix Functions
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913120165822464 |
|---|---|
| author | Singh, Navjot Barros, Kipton Li, Xiaoye Sherry |
| author_facet | Singh, Navjot Barros, Kipton Li, Xiaoye Sherry |
| contents | Objectives involving bilinear forms $u^\top f(A(θ))v$ for Hermitian $A$ arise widely in scientific computing and probabilistic machine learning. For large matrices, Lanczos efficiently approximates these quantities, but differentiating them with respect to $θ$ is challenging. Existing approaches either backpropagate through the Lanczos recurrence, requiring reorthogonalization for stability, or apply Arnoldi to an augmented block matrix of twice the original size. Both introduce extra computation and orthogonalization costs that can limit performance on modern hardware. We propose a forward-only gradient approximation that reuses the Lanczos pass and adds very minimal overhead in most cases. We prove that its error is proportional to the Lanczos residual norm, the same quantity controlling the forward approximation. Whereas a traditional adjoint-based calculation would be unstable without reorthogonalization, the new method appears unconditionally stable in our tests. It is also faster than existing state-of-the-art approaches. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_12801 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Fast and Stable Gradient Approximation for Bilinear Forms of Hermitian Matrix Functions Singh, Navjot Barros, Kipton Li, Xiaoye Sherry Numerical Analysis Objectives involving bilinear forms $u^\top f(A(θ))v$ for Hermitian $A$ arise widely in scientific computing and probabilistic machine learning. For large matrices, Lanczos efficiently approximates these quantities, but differentiating them with respect to $θ$ is challenging. Existing approaches either backpropagate through the Lanczos recurrence, requiring reorthogonalization for stability, or apply Arnoldi to an augmented block matrix of twice the original size. Both introduce extra computation and orthogonalization costs that can limit performance on modern hardware. We propose a forward-only gradient approximation that reuses the Lanczos pass and adds very minimal overhead in most cases. We prove that its error is proportional to the Lanczos residual norm, the same quantity controlling the forward approximation. Whereas a traditional adjoint-based calculation would be unstable without reorthogonalization, the new method appears unconditionally stable in our tests. It is also faster than existing state-of-the-art approaches. |
| title | Fast and Stable Gradient Approximation for Bilinear Forms of Hermitian Matrix Functions |
| topic | Numerical Analysis |
| url | https://arxiv.org/abs/2605.12801 |