Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2405.17248 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908352146046976 |
|---|---|
| author | Wang, Aaron T. Convertino, William Cheng, Xiang Henao, Ricardo Carin, Lawrence |
| author_facet | Wang, Aaron T. Convertino, William Cheng, Xiang Henao, Ricardo Carin, Lawrence |
| contents | In-context learning based on attention models is examined for data with categorical outcomes, with inference in such models viewed from the perspective of functional gradient descent (GD). We develop a network composed of attention blocks, with each block employing a self-attention layer followed by a cross-attention layer, with associated skip connections. This model can exactly perform multi-step functional GD inference for in-context inference with categorical observations. We perform a theoretical analysis of this setup, generalizing many prior assumptions in this line of work, including the class of attention mechanisms for which it is appropriate. We demonstrate the framework empirically on synthetic data, image classification and language generation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_17248 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | On Understanding Attention-Based In-Context Learning for Categorical Data Wang, Aaron T. Convertino, William Cheng, Xiang Henao, Ricardo Carin, Lawrence Machine Learning In-context learning based on attention models is examined for data with categorical outcomes, with inference in such models viewed from the perspective of functional gradient descent (GD). We develop a network composed of attention blocks, with each block employing a self-attention layer followed by a cross-attention layer, with associated skip connections. This model can exactly perform multi-step functional GD inference for in-context inference with categorical observations. We perform a theoretical analysis of this setup, generalizing many prior assumptions in this line of work, including the class of attention mechanisms for which it is appropriate. We demonstrate the framework empirically on synthetic data, image classification and language generation. |
| title | On Understanding Attention-Based In-Context Learning for Categorical Data |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2405.17248 |