Efficient Bayesian Updates for Deep Active Learning via Laplace Approximations
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2022
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866915852371099648 |
|---|---|
| author | Huseljic, Denis Herde, Marek Rauch, Lukas Hahn, Paul Huang, Zhixin Kottke, Daniel Vogt, Stephan Sick, Bernhard |
| author_facet | Huseljic, Denis Herde, Marek Rauch, Lukas Hahn, Paul Huang, Zhixin Kottke, Daniel Vogt, Stephan Sick, Bernhard |
| contents | Deep active learning (AL) selects batches of instances for annotation to avoid retraining deep neural networks (DNNs) after each new label. Employing a naive top-$b$ selection can result in a batch of redundant (similar) instances. To address this, various AL strategies employ clustering techniques that ensure diversity within a batch. We approach this issue by substituting the costly retraining with an efficient Bayesian update. Our proposed update represents a second-order optimization step using the Gaussian posterior from a last-layer Laplace approximation. Thereby, we achieve low computational complexity by computing the inverse Hessian in closed form. We demonstrate that in typical AL settings, our update closely approximates retraining while being considerably faster. Leveraging our update, we introduce a new framework for batch selection through sequential construction, updating the DNN after each label acquisition. Furthermore, we incorporate our update into a look-ahead selection strategy as a feasible upper baseline approximating optimal batch selection. Our results highlight the potential of efficient updates to advance deep AL research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2210_06112 |
| institution | arXiv |
| publishDate | 2022 |
| record_format | arxiv |
| spellingShingle | Efficient Bayesian Updates for Deep Active Learning via Laplace Approximations Huseljic, Denis Herde, Marek Rauch, Lukas Hahn, Paul Huang, Zhixin Kottke, Daniel Vogt, Stephan Sick, Bernhard Machine Learning Deep active learning (AL) selects batches of instances for annotation to avoid retraining deep neural networks (DNNs) after each new label. Employing a naive top-$b$ selection can result in a batch of redundant (similar) instances. To address this, various AL strategies employ clustering techniques that ensure diversity within a batch. We approach this issue by substituting the costly retraining with an efficient Bayesian update. Our proposed update represents a second-order optimization step using the Gaussian posterior from a last-layer Laplace approximation. Thereby, we achieve low computational complexity by computing the inverse Hessian in closed form. We demonstrate that in typical AL settings, our update closely approximates retraining while being considerably faster. Leveraging our update, we introduce a new framework for batch selection through sequential construction, updating the DNN after each label acquisition. Furthermore, we incorporate our update into a look-ahead selection strategy as a feasible upper baseline approximating optimal batch selection. Our results highlight the potential of efficient updates to advance deep AL research. |
| title | Efficient Bayesian Updates for Deep Active Learning via Laplace Approximations |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2210.06112 |