Provable Length Generalization in Sequence Prediction via Spectral Filtering
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916465853071360 |
|---|---|
| author | Marsden, Annie Dogariu, Evan Agarwal, Naman Chen, Xinyi Suo, Daniel Hazan, Elad |
| author_facet | Marsden, Annie Dogariu, Evan Agarwal, Naman Chen, Xinyi Suo, Daniel Hazan, Elad |
| contents | We consider the problem of length generalization in sequence prediction. We define a new metric of performance in this setting -- the Asymmetric-Regret -- which measures regret against a benchmark predictor with longer context length than available to the learner. We continue by studying this concept through the lens of the spectral filtering algorithm. We present a gradient-based learning algorithm that provably achieves length generalization for linear dynamical systems. We conclude with proof-of-concept experiments which are consistent with our theory. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_01035 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Provable Length Generalization in Sequence Prediction via Spectral Filtering Marsden, Annie Dogariu, Evan Agarwal, Naman Chen, Xinyi Suo, Daniel Hazan, Elad Machine Learning Artificial Intelligence Computation and Language We consider the problem of length generalization in sequence prediction. We define a new metric of performance in this setting -- the Asymmetric-Regret -- which measures regret against a benchmark predictor with longer context length than available to the learner. We continue by studying this concept through the lens of the spectral filtering algorithm. We present a gradient-based learning algorithm that provably achieves length generalization for linear dynamical systems. We conclude with proof-of-concept experiments which are consistent with our theory. |
| title | Provable Length Generalization in Sequence Prediction via Spectral Filtering |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2411.01035 |