LoRA-Ensemble: Efficient Uncertainty Modelling for Self-Attention Networks
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910044209020928 |
|---|---|
| author | Mühlematter, Dominik J. Halbheer, Michelle Becker, Alexander Narnhofer, Dominik Aasen, Helge Schindler, Konrad Turkoglu, Mehmet Ozgur |
| author_facet | Mühlematter, Dominik J. Halbheer, Michelle Becker, Alexander Narnhofer, Dominik Aasen, Helge Schindler, Konrad Turkoglu, Mehmet Ozgur |
| contents | Numerous real-world decisions rely on machine learning algorithms and require calibrated uncertainty estimates. However, modern methods often yield overconfident, uncalibrated predictions. The dominant approach to quantifying the uncertainty inherent in the model is to train an ensemble of separate predictors and measure their empirical variance. In an explicit implementation, the ensemble has a high computational cost and memory footprint, especially if the base model itself is already large, like modern transformers. This motivates efforts to develop implicit ensemble methods that emulate the ensemble without explicitly instantiating all its members. We introduce LoRA-Ensemble, a parameter-efficient ensembling method for self-attention networks. It is based on Low-Rank Adaptation (LoRA), originally developed for efficient LLM fine-tuning, and extends it into an implicit ensembling scheme, where all ensemble members share the same, pre-trained self-attention network, but have individual low-rank matrices for the attention projections. The resulting method not only outperforms state-of-the-art implicit techniques like BatchEnsemble, but even matches or exceeds the accuracy of an Explicit Ensemble, while at the same time achieving superior calibration. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_14438 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LoRA-Ensemble: Efficient Uncertainty Modelling for Self-Attention Networks Mühlematter, Dominik J. Halbheer, Michelle Becker, Alexander Narnhofer, Dominik Aasen, Helge Schindler, Konrad Turkoglu, Mehmet Ozgur Machine Learning Numerous real-world decisions rely on machine learning algorithms and require calibrated uncertainty estimates. However, modern methods often yield overconfident, uncalibrated predictions. The dominant approach to quantifying the uncertainty inherent in the model is to train an ensemble of separate predictors and measure their empirical variance. In an explicit implementation, the ensemble has a high computational cost and memory footprint, especially if the base model itself is already large, like modern transformers. This motivates efforts to develop implicit ensemble methods that emulate the ensemble without explicitly instantiating all its members. We introduce LoRA-Ensemble, a parameter-efficient ensembling method for self-attention networks. It is based on Low-Rank Adaptation (LoRA), originally developed for efficient LLM fine-tuning, and extends it into an implicit ensembling scheme, where all ensemble members share the same, pre-trained self-attention network, but have individual low-rank matrices for the attention projections. The resulting method not only outperforms state-of-the-art implicit techniques like BatchEnsemble, but even matches or exceeds the accuracy of an Explicit Ensemble, while at the same time achieving superior calibration. |
| title | LoRA-Ensemble: Efficient Uncertainty Modelling for Self-Attention Networks |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2405.14438 |