Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2501.15556 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910801413013504 |
|---|---|
| author | Rukhovich, Alexey Podolskiy, Alexander Piontkovskaya, Irina |
| author_facet | Rukhovich, Alexey Podolskiy, Alexander Piontkovskaya, Irina |
| contents | In multi-domain learning, a single model is trained on diverse data domains to leverage shared knowledge and improve generalization. The order in which the data from these domains is used for training can significantly affect the model's performance on each domain. However, this dependence is under-studied. In this paper, we investigate the influence of training order (or data mixing) in multi-domain learning using the concept of Lie bracket of gradient vector fields. By analyzing the infinitesimal effects of changing the training order, we identify regions in the parameter space where altering the order between two training domains can benefit the target loss. We validate the predictions of our theoretical framework on the influence of training order (or data mixing) both on a toy example and bilingual LLM pre-training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_15556 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Commute Your Domains: Trajectory Optimality Criterion for Multi-Domain Learning Rukhovich, Alexey Podolskiy, Alexander Piontkovskaya, Irina Machine Learning Computation and Language In multi-domain learning, a single model is trained on diverse data domains to leverage shared knowledge and improve generalization. The order in which the data from these domains is used for training can significantly affect the model's performance on each domain. However, this dependence is under-studied. In this paper, we investigate the influence of training order (or data mixing) in multi-domain learning using the concept of Lie bracket of gradient vector fields. By analyzing the infinitesimal effects of changing the training order, we identify regions in the parameter space where altering the order between two training domains can benefit the target loss. We validate the predictions of our theoretical framework on the influence of training order (or data mixing) both on a toy example and bilingual LLM pre-training. |
| title | Commute Your Domains: Trajectory Optimality Criterion for Multi-Domain Learning |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2501.15556 |