Alternate Loss Functions for Classification and Robust Regression Can Improve the Accuracy of Artificial Neural Networks
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917828243750912 |
|---|---|
| author | Noel, Mathew Mithra Banerjee, Arindam Oswal, Yug D, Geraldine Bessie Amali Muthiah-Nakarajan, Venkataraman |
| author_facet | Noel, Mathew Mithra Banerjee, Arindam Oswal, Yug D, Geraldine Bessie Amali Muthiah-Nakarajan, Venkataraman |
| contents | All machine learning algorithms use a loss, cost, utility or reward function to encode the learning objective and oversee the learning process. This function that supervises learning is a frequently unrecognized hyperparameter that determines how incorrect outputs are penalized and can be tuned to improve performance. This paper shows that training speed and final accuracy of neural networks can significantly depend on the loss function used to train neural networks. In particular derivative values can be significantly different with different loss functions leading to significantly different performance after gradient descent based Backpropagation (BP) training. This paper explores the effect on performance of using new loss functions that are also convex but penalize errors differently compared to the popular Cross-entropy loss. Two new classification loss functions that significantly improve performance on a wide variety of benchmark tasks are proposed. A new loss function call smooth absolute error that outperforms the Squared error, Huber and Log-Cosh losses on datasets with significantly many outliers is proposed. This smooth absolute error loss function is infinitely differentiable and more closely approximates the absolute error loss compared to the Huber and Log-Cosh losses used for robust regression. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2303_09935 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Alternate Loss Functions for Classification and Robust Regression Can Improve the Accuracy of Artificial Neural Networks Noel, Mathew Mithra Banerjee, Arindam Oswal, Yug D, Geraldine Bessie Amali Muthiah-Nakarajan, Venkataraman Neural and Evolutionary Computing 68 I.2 All machine learning algorithms use a loss, cost, utility or reward function to encode the learning objective and oversee the learning process. This function that supervises learning is a frequently unrecognized hyperparameter that determines how incorrect outputs are penalized and can be tuned to improve performance. This paper shows that training speed and final accuracy of neural networks can significantly depend on the loss function used to train neural networks. In particular derivative values can be significantly different with different loss functions leading to significantly different performance after gradient descent based Backpropagation (BP) training. This paper explores the effect on performance of using new loss functions that are also convex but penalize errors differently compared to the popular Cross-entropy loss. Two new classification loss functions that significantly improve performance on a wide variety of benchmark tasks are proposed. A new loss function call smooth absolute error that outperforms the Squared error, Huber and Log-Cosh losses on datasets with significantly many outliers is proposed. This smooth absolute error loss function is infinitely differentiable and more closely approximates the absolute error loss compared to the Huber and Log-Cosh losses used for robust regression. |
| title | Alternate Loss Functions for Classification and Robust Regression Can Improve the Accuracy of Artificial Neural Networks |
| topic | Neural and Evolutionary Computing 68 I.2 |
| url | https://arxiv.org/abs/2303.09935 |