Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2402.00501 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909361102651392 |
|---|---|
| author | Daunas, Francisco Esnaola, Iñaki Perlaza, Samir M. Poor, H. Vincent |
| author_facet | Daunas, Francisco Esnaola, Iñaki Perlaza, Samir M. Poor, H. Vincent |
| contents | The solution to empirical risk minimization with $f$-divergence regularization (ERM-$f$DR) is presented under mild conditions on $f$. Under such conditions, the optimal measure is shown to be unique. Examples of the solution for particular choices of the function $f$ are presented. Previously known solutions to common regularization choices are obtained by leveraging the flexibility of the family of $f$-divergences. These include the unique solutions to empirical risk minimization with relative entropy regularization (Type-I and Type-II). The analysis of the solution unveils the following properties of $f$-divergences when used in the ERM-$f$DR problem: $i\bigl)$ $f$-divergence regularization forces the support of the solution to coincide with the support of the reference measure, which introduces a strong inductive bias that dominates the evidence provided by the training data; and $ii\bigl)$ any $f$-divergence regularization is equivalent to a different $f$-divergence regularization with an appropriate transformation of the empirical risk function. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_00501 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Equivalence of the Empirical Risk Minimization to Regularization on the Family of f-Divergences Daunas, Francisco Esnaola, Iñaki Perlaza, Samir M. Poor, H. Vincent Machine Learning Information Theory The solution to empirical risk minimization with $f$-divergence regularization (ERM-$f$DR) is presented under mild conditions on $f$. Under such conditions, the optimal measure is shown to be unique. Examples of the solution for particular choices of the function $f$ are presented. Previously known solutions to common regularization choices are obtained by leveraging the flexibility of the family of $f$-divergences. These include the unique solutions to empirical risk minimization with relative entropy regularization (Type-I and Type-II). The analysis of the solution unveils the following properties of $f$-divergences when used in the ERM-$f$DR problem: $i\bigl)$ $f$-divergence regularization forces the support of the solution to coincide with the support of the reference measure, which introduces a strong inductive bias that dominates the evidence provided by the training data; and $ii\bigl)$ any $f$-divergence regularization is equivalent to a different $f$-divergence regularization with an appropriate transformation of the empirical risk function. |
| title | Equivalence of the Empirical Risk Minimization to Regularization on the Family of f-Divergences |
| topic | Machine Learning Information Theory |
| url | https://arxiv.org/abs/2402.00501 |