Hessian of Perplexity for Large Language Models by PyTorch autograd (Open Source)
Fuente:
arXiv
Saved in:
| Main Author: | |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913780223442944 |
|---|---|
| author | Ilin, Ivan |
| author_facet | Ilin, Ivan |
| contents | Computing the full Hessian matrix -- the matrix of second-order derivatives for an entire Large Language Model (LLM) is infeasible due to its sheer size. In this technical report, we aim to provide a comprehensive guide on how to accurately compute at least a small portion of the Hessian for LLMs using PyTorch autograd library. We also demonstrate how to compute the full diagonal of the Hessian matrix using multiple samples of vector-Hessian Products (HVPs). We hope that both this guide and the accompanying GitHub code will be valuable resources for practitioners and researchers interested in better understanding the behavior and structure of the Hessian in LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_04520 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Hessian of Perplexity for Large Language Models by PyTorch autograd (Open Source) Ilin, Ivan Machine Learning Artificial Intelligence Computation and Language 68T07, 65K10, 65Y05 Computing the full Hessian matrix -- the matrix of second-order derivatives for an entire Large Language Model (LLM) is infeasible due to its sheer size. In this technical report, we aim to provide a comprehensive guide on how to accurately compute at least a small portion of the Hessian for LLMs using PyTorch autograd library. We also demonstrate how to compute the full diagonal of the Hessian matrix using multiple samples of vector-Hessian Products (HVPs). We hope that both this guide and the accompanying GitHub code will be valuable resources for practitioners and researchers interested in better understanding the behavior and structure of the Hessian in LLMs. |
| title | Hessian of Perplexity for Large Language Models by PyTorch autograd (Open Source) |
| topic | Machine Learning Artificial Intelligence Computation and Language 68T07, 65K10, 65Y05 |
| url | https://arxiv.org/abs/2504.04520 |