Better Training Data Attribution via Better Inverse Hessian-Vector Products
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912492812238848 |
|---|---|
| author | Wang, Andrew Nguyen, Elisa Yang, Runshi Bae, Juhan McIlraith, Sheila A. Grosse, Roger |
| author_facet | Wang, Andrew Nguyen, Elisa Yang, Runshi Bae, Juhan McIlraith, Sheila A. Grosse, Roger |
| contents | Training data attribution (TDA) provides insights into which training data is responsible for a learned model behavior. Gradient-based TDA methods such as influence functions and unrolled differentiation both involve a computation that resembles an inverse Hessian-vector product (iHVP), which is difficult to approximate efficiently. We introduce an algorithm (ASTRA) which uses the EKFAC-preconditioner on Neumann series iterations to arrive at an accurate iHVP approximation for TDA. ASTRA is easy to tune, requires fewer iterations than Neumann series iterations, and is more accurate than EKFAC-based approximations. Using ASTRA, we show that improving the accuracy of the iHVP approximation can significantly improve TDA performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_14740 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Better Training Data Attribution via Better Inverse Hessian-Vector Products Wang, Andrew Nguyen, Elisa Yang, Runshi Bae, Juhan McIlraith, Sheila A. Grosse, Roger Machine Learning Training data attribution (TDA) provides insights into which training data is responsible for a learned model behavior. Gradient-based TDA methods such as influence functions and unrolled differentiation both involve a computation that resembles an inverse Hessian-vector product (iHVP), which is difficult to approximate efficiently. We introduce an algorithm (ASTRA) which uses the EKFAC-preconditioner on Neumann series iterations to arrive at an accurate iHVP approximation for TDA. ASTRA is easy to tune, requires fewer iterations than Neumann series iterations, and is more accurate than EKFAC-based approximations. Using ASTRA, we show that improving the accuracy of the iHVP approximation can significantly improve TDA performance. |
| title | Better Training Data Attribution via Better Inverse Hessian-Vector Products |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2507.14740 |