HatLLM: Hierarchical Attention Masking for Enhanced Collaborative Modeling in LLM-based Recommendation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915549217292288 |
|---|---|
| author | Cui, Yu Liu, Feng Chen, Jiawei Jin, Canghong Lou, Xingyu Zhang, Changwang Wang, Jun Sun, Yuegang Wang, Can |
| author_facet | Cui, Yu Liu, Feng Chen, Jiawei Jin, Canghong Lou, Xingyu Zhang, Changwang Wang, Jun Sun, Yuegang Wang, Can |
| contents | Recent years have witnessed a surge of research on leveraging large language models (LLMs) for sequential recommendation. LLMs have demonstrated remarkable potential in inferring users' nuanced preferences through fine-grained semantic reasoning. However, they also exhibit a notable limitation in effectively modeling collaborative signals, i.e., behavioral correlations inherent in users' historical interactions. Our empirical analysis further reveals that the attention mechanisms in LLMs tend to disproportionately focus on tokens within the same item, thereby impeding the capture of cross-item correlations.
To address this limitation, we propose a novel hierarchical attention masking strategy for LLM-based recommendation, termed HatLLM. Specifically, in shallow layers, HatLLM masks attention between tokens from different items, facilitating intra-item semantic understanding; in contrast, in deep layers, HatLLM masks attention within items, thereby compelling the model to capture cross-item correlations. This progressive, layer-wise approach enables LLMs to jointly model both token-level and item-level dependencies. Extensive experiments on three real-world datasets demonstrate that HatLLM achieves significant performance gains (9.13% on average) over existing LLM-based methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_10955 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | HatLLM: Hierarchical Attention Masking for Enhanced Collaborative Modeling in LLM-based Recommendation Cui, Yu Liu, Feng Chen, Jiawei Jin, Canghong Lou, Xingyu Zhang, Changwang Wang, Jun Sun, Yuegang Wang, Can Information Retrieval Recent years have witnessed a surge of research on leveraging large language models (LLMs) for sequential recommendation. LLMs have demonstrated remarkable potential in inferring users' nuanced preferences through fine-grained semantic reasoning. However, they also exhibit a notable limitation in effectively modeling collaborative signals, i.e., behavioral correlations inherent in users' historical interactions. Our empirical analysis further reveals that the attention mechanisms in LLMs tend to disproportionately focus on tokens within the same item, thereby impeding the capture of cross-item correlations. To address this limitation, we propose a novel hierarchical attention masking strategy for LLM-based recommendation, termed HatLLM. Specifically, in shallow layers, HatLLM masks attention between tokens from different items, facilitating intra-item semantic understanding; in contrast, in deep layers, HatLLM masks attention within items, thereby compelling the model to capture cross-item correlations. This progressive, layer-wise approach enables LLMs to jointly model both token-level and item-level dependencies. Extensive experiments on three real-world datasets demonstrate that HatLLM achieves significant performance gains (9.13% on average) over existing LLM-based methods. |
| title | HatLLM: Hierarchical Attention Masking for Enhanced Collaborative Modeling in LLM-based Recommendation |
| topic | Information Retrieval |
| url | https://arxiv.org/abs/2510.10955 |