Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2411.05052 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910689272004608 |
|---|---|
| author | Bleiberg, Yair Werman, Michael |
| author_facet | Bleiberg, Yair Werman, Michael |
| contents | Coordinate-based Multi-Layer Perceptrons (MLPs) are known to have difficulty reconstructing high frequencies of the training data. A common solution to this problem is Positional Encoding (PE), which has become quite popular. However, PE has drawbacks. It has high-frequency artifacts and adds another hyper-hyperparameter, just like batch normalization and dropout do. We believe that under certain circumstances PE is not necessary, and a smarter construction of the network architecture together with a smart training method is sufficient to achieve similar results. In this paper, we show that very simple MLPs can quite easily output a frequency when given input of the half-frequency and quarter-frequency. Using this, we design a network architecture in blocks, where the input to each block is the output of the two previous blocks along with the original input. We call this a {\it Fibonacci Network}. By training each block on the corresponding frequencies of the signal, we show that Fibonacci Networks can reconstruct arbitrarily high frequencies. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_05052 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | The Fibonacci Network: A Simple Alternative for Positional Encoding Bleiberg, Yair Werman, Michael Machine Learning Coordinate-based Multi-Layer Perceptrons (MLPs) are known to have difficulty reconstructing high frequencies of the training data. A common solution to this problem is Positional Encoding (PE), which has become quite popular. However, PE has drawbacks. It has high-frequency artifacts and adds another hyper-hyperparameter, just like batch normalization and dropout do. We believe that under certain circumstances PE is not necessary, and a smarter construction of the network architecture together with a smart training method is sufficient to achieve similar results. In this paper, we show that very simple MLPs can quite easily output a frequency when given input of the half-frequency and quarter-frequency. Using this, we design a network architecture in blocks, where the input to each block is the output of the two previous blocks along with the original input. We call this a {\it Fibonacci Network}. By training each block on the corresponding frequencies of the signal, we show that Fibonacci Networks can reconstruct arbitrarily high frequencies. |
| title | The Fibonacci Network: A Simple Alternative for Positional Encoding |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2411.05052 |