Polynomial Regression as a Task for Understanding In-context Learning Through Finetuning and Alignment
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929439100633088 |
|---|---|
| author | Wilcoxson, Max Svendgård, Morten Doshi, Ria Davis, Dylan Vir, Reya Sahai, Anant |
| author_facet | Wilcoxson, Max Svendgård, Morten Doshi, Ria Davis, Dylan Vir, Reya Sahai, Anant |
| contents | Simple function classes have emerged as toy problems to better understand in-context-learning in transformer-based architectures used for large language models. But previously proposed simple function classes like linear regression or multi-layer-perceptrons lack the structure required to explore things like prompting and alignment within models capable of in-context-learning. We propose univariate polynomial regression as a function class that is just rich enough to study prompting and alignment, while allowing us to visualize and understand what is going on clearly. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_19346 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Polynomial Regression as a Task for Understanding In-context Learning Through Finetuning and Alignment Wilcoxson, Max Svendgård, Morten Doshi, Ria Davis, Dylan Vir, Reya Sahai, Anant Machine Learning Computation and Language Simple function classes have emerged as toy problems to better understand in-context-learning in transformer-based architectures used for large language models. But previously proposed simple function classes like linear regression or multi-layer-perceptrons lack the structure required to explore things like prompting and alignment within models capable of in-context-learning. We propose univariate polynomial regression as a function class that is just rich enough to study prompting and alignment, while allowing us to visualize and understand what is going on clearly. |
| title | Polynomial Regression as a Task for Understanding In-context Learning Through Finetuning and Alignment |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2407.19346 |