Harmonizing Program Induction with Rate-Distortion Theory
Fuente:
arXiv
Salvato in:
| Autori principali: | , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917662218518528 |
|---|---|
| author | Zhou, Hanqi Nagy, David G. Wu, Charley M. |
| author_facet | Zhou, Hanqi Nagy, David G. Wu, Charley M. |
| contents | Many aspects of human learning have been proposed as a process of constructing mental programs: from acquiring symbolic number representations to intuitive theories about the world. In parallel, there is a long-tradition of using information processing to model human cognition through Rate Distortion Theory (RDT). Yet, it is still poorly understood how to apply RDT when mental representations take the form of programs. In this work, we adapt RDT by proposing a three way trade-off among rate (description length), distortion (error), and computational costs (search budget). We use simulations on a melody task to study the implications of this trade-off, and show that constructing a shared program library across tasks provides global benefits. However, this comes at the cost of sensitivity to curricula, which is also characteristic of human learners. Finally, we use methods from partial information decomposition to generate training curricula that induce more effective libraries and better generalization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_05294 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Harmonizing Program Induction with Rate-Distortion Theory Zhou, Hanqi Nagy, David G. Wu, Charley M. Human-Computer Interaction Computation and Language Information Theory Machine Learning Symbolic Computation Many aspects of human learning have been proposed as a process of constructing mental programs: from acquiring symbolic number representations to intuitive theories about the world. In parallel, there is a long-tradition of using information processing to model human cognition through Rate Distortion Theory (RDT). Yet, it is still poorly understood how to apply RDT when mental representations take the form of programs. In this work, we adapt RDT by proposing a three way trade-off among rate (description length), distortion (error), and computational costs (search budget). We use simulations on a melody task to study the implications of this trade-off, and show that constructing a shared program library across tasks provides global benefits. However, this comes at the cost of sensitivity to curricula, which is also characteristic of human learners. Finally, we use methods from partial information decomposition to generate training curricula that induce more effective libraries and better generalization. |
| title | Harmonizing Program Induction with Rate-Distortion Theory |
| topic | Human-Computer Interaction Computation and Language Information Theory Machine Learning Symbolic Computation |
| url | https://arxiv.org/abs/2405.05294 |