Temporal Task Diversity: Inductive Biases Under Non-Stationarity in Synthetic Sequence Modelling
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914577929732096 |
|---|---|
| author | Aswadi, Afiq Abdillah Effiezal Britton, Oliver Baker, Ross Farrugia-Roberts, Matthew |
| author_facet | Aswadi, Afiq Abdillah Effiezal Britton, Oliver Baker, Ross Farrugia-Roberts, Matthew |
| contents | Modern deep learning science often assumes that neural networks learn from a fixed data distribution. However, many practically important learning problems involve data distributions that change throughout training. How does such non-stationarity impact the inductive biases of deep learning towards models with different structural, generalisation, and safety properties? A fruitful testbed for studying inductive bias is in-context linear regression sequence modelling, where small transformers display strikingly different generalisation patterns depending on the diversity of the (fixed) training task distribution. In this paper, we explore the effect of diversifying the task distribution across training time, finding that such temporal diversity leads to an increased bias towards generalisation over memorisation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_18281 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Temporal Task Diversity: Inductive Biases Under Non-Stationarity in Synthetic Sequence Modelling Aswadi, Afiq Abdillah Effiezal Britton, Oliver Baker, Ross Farrugia-Roberts, Matthew Machine Learning Modern deep learning science often assumes that neural networks learn from a fixed data distribution. However, many practically important learning problems involve data distributions that change throughout training. How does such non-stationarity impact the inductive biases of deep learning towards models with different structural, generalisation, and safety properties? A fruitful testbed for studying inductive bias is in-context linear regression sequence modelling, where small transformers display strikingly different generalisation patterns depending on the diversity of the (fixed) training task distribution. In this paper, we explore the effect of diversifying the task distribution across training time, finding that such temporal diversity leads to an increased bias towards generalisation over memorisation. |
| title | Temporal Task Diversity: Inductive Biases Under Non-Stationarity in Synthetic Sequence Modelling |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2605.18281 |