RanDeS: Randomized Delta Superposition for Multi-Model Compression
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910947878109184 |
|---|---|
| author | Zhou, Hangyu Gokaslan, Aaron Kuleshov, Volodymyr Hariharan, Bharath |
| author_facet | Zhou, Hangyu Gokaslan, Aaron Kuleshov, Volodymyr Hariharan, Bharath |
| contents | From a multi-model compression perspective, model merging enables memory-efficient serving of multiple models fine-tuned from the same base, but suffers from degraded performance due to interference among their task-specific parameter adjustments (i.e., deltas). In this paper, we reformulate model merging as a compress-and-retrieve scheme, revealing that the task interference arises from the summation of irrelevant deltas during model retrieval. To address this issue, we use random orthogonal transformations to decorrelate these vectors into self-cancellation. We show that this approach drastically reduces interference, improving performance across both vision and language tasks. Since these transformations are fully defined by random seeds, adding new models requires no extra memory. Further, their data- and model-agnostic nature enables easy addition or removal of models with minimal compute overhead, supporting efficient and flexible multi-model serving. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_11204 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | RanDeS: Randomized Delta Superposition for Multi-Model Compression Zhou, Hangyu Gokaslan, Aaron Kuleshov, Volodymyr Hariharan, Bharath Machine Learning Artificial Intelligence From a multi-model compression perspective, model merging enables memory-efficient serving of multiple models fine-tuned from the same base, but suffers from degraded performance due to interference among their task-specific parameter adjustments (i.e., deltas). In this paper, we reformulate model merging as a compress-and-retrieve scheme, revealing that the task interference arises from the summation of irrelevant deltas during model retrieval. To address this issue, we use random orthogonal transformations to decorrelate these vectors into self-cancellation. We show that this approach drastically reduces interference, improving performance across both vision and language tasks. Since these transformations are fully defined by random seeds, adding new models requires no extra memory. Further, their data- and model-agnostic nature enables easy addition or removal of models with minimal compute overhead, supporting efficient and flexible multi-model serving. |
| title | RanDeS: Randomized Delta Superposition for Multi-Model Compression |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2505.11204 |