Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2501.07596 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912256553385984 |
|---|---|
| author | Lv, Zheqi Ye, Keming Wei, Zishu Tian, Qi Zhang, Shengyu Zhang, Wenqiao Wang, Wenjie Kuang, Kun Chua, Tat-Seng Wu, Fei |
| author_facet | Lv, Zheqi Ye, Keming Wei, Zishu Tian, Qi Zhang, Shengyu Zhang, Wenqiao Wang, Wenjie Kuang, Kun Chua, Tat-Seng Wu, Fei |
| contents | Deep neural networks have become foundational to advancements in multiple domains, including recommendation systems, natural language processing, and so on. Despite their successes, these models often contain incompatible parameters that can be underutilized or detrimental to model performance, particularly when faced with specific, varying data distributions. Existing research excels in removing such parameters or merging the outputs of multiple different pretrained models. However, the former focuses on efficiency rather than performance, while the latter requires several times more computing and storage resources to support inference. In this paper, we set the goal to explicitly improve these incompatible parameters by leveraging the complementary strengths of different models, thereby directly enhancing the models without any additional parameters. Specifically, we propose Compatibility-aware Knowledge Integration (CKI), which consists of Parameter Compatibility Assessment and Parameter Splicing, which are used to evaluate the knowledge content of multiple models and integrate the knowledge into one model, respectively. The integrated model can be used directly for inference or for further fine-tuning. We conduct extensive experiments on various datasets for recommendation and language tasks, and the results show that Compatibility-aware Knowledge Integration can effectively optimize incompatible parameters under multiple tasks and settings to break through the training limit of the original model without increasing the inference cost. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_07596 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Optimize Incompatible Parameters through Compatibility-aware Knowledge Integration Lv, Zheqi Ye, Keming Wei, Zishu Tian, Qi Zhang, Shengyu Zhang, Wenqiao Wang, Wenjie Kuang, Kun Chua, Tat-Seng Wu, Fei Machine Learning Computation and Language Information Retrieval Deep neural networks have become foundational to advancements in multiple domains, including recommendation systems, natural language processing, and so on. Despite their successes, these models often contain incompatible parameters that can be underutilized or detrimental to model performance, particularly when faced with specific, varying data distributions. Existing research excels in removing such parameters or merging the outputs of multiple different pretrained models. However, the former focuses on efficiency rather than performance, while the latter requires several times more computing and storage resources to support inference. In this paper, we set the goal to explicitly improve these incompatible parameters by leveraging the complementary strengths of different models, thereby directly enhancing the models without any additional parameters. Specifically, we propose Compatibility-aware Knowledge Integration (CKI), which consists of Parameter Compatibility Assessment and Parameter Splicing, which are used to evaluate the knowledge content of multiple models and integrate the knowledge into one model, respectively. The integrated model can be used directly for inference or for further fine-tuning. We conduct extensive experiments on various datasets for recommendation and language tasks, and the results show that Compatibility-aware Knowledge Integration can effectively optimize incompatible parameters under multiple tasks and settings to break through the training limit of the original model without increasing the inference cost. |
| title | Optimize Incompatible Parameters through Compatibility-aware Knowledge Integration |
| topic | Machine Learning Computation and Language Information Retrieval |
| url | https://arxiv.org/abs/2501.07596 |