Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.09435 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911202672640000 |
|---|---|
| author | Lee, Hyunin Zhang, Yong Nguyen, Hoang Vu Liu, Xiaoyi Park, Namyong Jung, Christopher Jin, Rong Wang, Yang Wang, Zhigang Sojoudi, Somayeh Feng, Xue |
| author_facet | Lee, Hyunin Zhang, Yong Nguyen, Hoang Vu Liu, Xiaoyi Park, Namyong Jung, Christopher Jin, Rong Wang, Yang Wang, Zhigang Sojoudi, Somayeh Feng, Xue |
| contents | Cross-domain sequential recommendation (CDSR) aims to align heterogeneous user behavior sequences collected from different domains. While cross-attention is widely used to enhance alignment and improve recommendation performance, its underlying mechanism is not fully understood. Most researchers interpret cross-attention as residual alignment, where the output is generated by removing redundant and preserving non-redundant information from the query input by referencing another domain data which is input key and value. Beyond the prevailing view, we introduce Orthogonal Alignment, a phenomenon in which cross-attention discovers novel information that is not present in the query input, and further argue that those two contrasting alignment mechanisms can co-exist in recommendation models We find that when the query input and output of cross-attention are orthogonal, model performance improves over 300 experiments. Notably, Orthogonal Alignment emerges naturally, without any explicit orthogonality constraints. Our key insight is that Orthogonal Alignment emerges naturally because it improves scaling law. We show that baselines additionally incorporating cross-attention module outperform parameter-matched baselines, achieving a superior accuracy-per-model parameter. We hope these findings offer new directions for parameter-efficient scaling in multi-modal research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_09435 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Cross-attention Secretly Performs Orthogonal Alignment in Recommendation Models Lee, Hyunin Zhang, Yong Nguyen, Hoang Vu Liu, Xiaoyi Park, Namyong Jung, Christopher Jin, Rong Wang, Yang Wang, Zhigang Sojoudi, Somayeh Feng, Xue Machine Learning Information Retrieval Cross-domain sequential recommendation (CDSR) aims to align heterogeneous user behavior sequences collected from different domains. While cross-attention is widely used to enhance alignment and improve recommendation performance, its underlying mechanism is not fully understood. Most researchers interpret cross-attention as residual alignment, where the output is generated by removing redundant and preserving non-redundant information from the query input by referencing another domain data which is input key and value. Beyond the prevailing view, we introduce Orthogonal Alignment, a phenomenon in which cross-attention discovers novel information that is not present in the query input, and further argue that those two contrasting alignment mechanisms can co-exist in recommendation models We find that when the query input and output of cross-attention are orthogonal, model performance improves over 300 experiments. Notably, Orthogonal Alignment emerges naturally, without any explicit orthogonality constraints. Our key insight is that Orthogonal Alignment emerges naturally because it improves scaling law. We show that baselines additionally incorporating cross-attention module outperform parameter-matched baselines, achieving a superior accuracy-per-model parameter. We hope these findings offer new directions for parameter-efficient scaling in multi-modal research. |
| title | Cross-attention Secretly Performs Orthogonal Alignment in Recommendation Models |
| topic | Machine Learning Information Retrieval |
| url | https://arxiv.org/abs/2510.09435 |