ORBIT: Preserving Foundational Language Capabilities in GenRetrieval via Origin-Regulated Merging
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910213243666432 |
|---|---|
| author | Verma, Neha Mehta, Nikhil Wang, Shao-Chuan Zhang, Naijing Tsai, Alicia Wei, Li Heldt, Lukasz Hong, Lichan Chi, Ed Yi, Xinyang |
| author_facet | Verma, Neha Mehta, Nikhil Wang, Shao-Chuan Zhang, Naijing Tsai, Alicia Wei, Li Heldt, Lukasz Hong, Lichan Chi, Ed Yi, Xinyang |
| contents | Despite the rapid advancements in large language model (LLM) development, fine-tuning them for specific tasks often results in the catastrophic forgetting of their general, language-based reasoning abilities. This work investigates and addresses this challenge in the context of the Generative Retrieval (GenRetrieval) task. During GenRetrieval fine-tuning, we find this forgetting occurs rapidly and correlates with the distance between the fine-tuned and original model parameters. Given these observations, we propose ORBIT, a novel approach that actively tracks the distance between fine-tuned and initial model weights, and uses a weight averaging strategy to constrain model drift during GenRetrieval fine-tuning when this inter-model distance exceeds a maximum threshold. Our results show that ORBIT retains substantial text and retrieval performance by outperforming both common continual learning baselines and related regularization methods that also employ weight averaging. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_12419 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | ORBIT: Preserving Foundational Language Capabilities in GenRetrieval via Origin-Regulated Merging Verma, Neha Mehta, Nikhil Wang, Shao-Chuan Zhang, Naijing Tsai, Alicia Wei, Li Heldt, Lukasz Hong, Lichan Chi, Ed Yi, Xinyang Computation and Language Information Retrieval Machine Learning Despite the rapid advancements in large language model (LLM) development, fine-tuning them for specific tasks often results in the catastrophic forgetting of their general, language-based reasoning abilities. This work investigates and addresses this challenge in the context of the Generative Retrieval (GenRetrieval) task. During GenRetrieval fine-tuning, we find this forgetting occurs rapidly and correlates with the distance between the fine-tuned and original model parameters. Given these observations, we propose ORBIT, a novel approach that actively tracks the distance between fine-tuned and initial model weights, and uses a weight averaging strategy to constrain model drift during GenRetrieval fine-tuning when this inter-model distance exceeds a maximum threshold. Our results show that ORBIT retains substantial text and retrieval performance by outperforming both common continual learning baselines and related regularization methods that also employ weight averaging. |
| title | ORBIT: Preserving Foundational Language Capabilities in GenRetrieval via Origin-Regulated Merging |
| topic | Computation and Language Information Retrieval Machine Learning |
| url | https://arxiv.org/abs/2605.12419 |