GRACE: Generative Recommendation via Journey-Aware Sparse Attention on Chain-of-Thought Tokenization
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912492894027776 |
|---|---|
| author | Ma, Luyi Zhang, Wanjia Zhao, Kai Kulkarni, Abhishek Morishetti, Lalitesh Ganesh, Anjana Ranjan, Ashish Padmanabhan, Aashika Xu, Jianpeng Cho, Jason Kanumala, Praveen Nag, Kaushiki Dutta, Sumit Motwani, Kamiya Patel, Malay Korpeoglu, Evren Kumar, Sushant Achan, Kannan |
| author_facet | Ma, Luyi Zhang, Wanjia Zhao, Kai Kulkarni, Abhishek Morishetti, Lalitesh Ganesh, Anjana Ranjan, Ashish Padmanabhan, Aashika Xu, Jianpeng Cho, Jason Kanumala, Praveen Nag, Kaushiki Dutta, Sumit Motwani, Kamiya Patel, Malay Korpeoglu, Evren Kumar, Sushant Achan, Kannan |
| contents | Generative models have recently demonstrated strong potential in multi-behavior recommendation systems, leveraging the expressive power of transformers and tokenization to generate personalized item sequences. However, their adoption is hindered by (1) the lack of explicit information for token reasoning, (2) high computational costs due to quadratic attention complexity and dense sequence representations after tokenization, and (3) limited multi-scale modeling over user history. In this work, we propose GRACE (Generative Recommendation via journey-aware sparse Attention on Chain-of-thought tokEnization), a novel generative framework for multi-behavior sequential recommendation. GRACE introduces a hybrid Chain-of-Thought (CoT) tokenization method that encodes user-item interactions with explicit attributes from product knowledge graphs (e.g., category, brand, price) over semantic tokenization, enabling interpretable and behavior-aligned generation. To address the inefficiency of standard attention, we design a Journey-Aware Sparse Attention (JSA) mechanism, which selectively attends to compressed, intra-, inter-, and current-context segments in the tokenized sequence. Experiments on two real-world datasets show that GRACE significantly outperforms state-of-the-art baselines, achieving up to +106.9% HR@10 and +106.7% NDCG@10 improvement over the state-of-the-art baseline on the Home domain, and +22.1% HR@10 on the Electronics domain. GRACE also reduces attention computation by up to 48% with long sequences. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_14758 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | GRACE: Generative Recommendation via Journey-Aware Sparse Attention on Chain-of-Thought Tokenization Ma, Luyi Zhang, Wanjia Zhao, Kai Kulkarni, Abhishek Morishetti, Lalitesh Ganesh, Anjana Ranjan, Ashish Padmanabhan, Aashika Xu, Jianpeng Cho, Jason Kanumala, Praveen Nag, Kaushiki Dutta, Sumit Motwani, Kamiya Patel, Malay Korpeoglu, Evren Kumar, Sushant Achan, Kannan Computation and Language Artificial Intelligence Information Retrieval Generative models have recently demonstrated strong potential in multi-behavior recommendation systems, leveraging the expressive power of transformers and tokenization to generate personalized item sequences. However, their adoption is hindered by (1) the lack of explicit information for token reasoning, (2) high computational costs due to quadratic attention complexity and dense sequence representations after tokenization, and (3) limited multi-scale modeling over user history. In this work, we propose GRACE (Generative Recommendation via journey-aware sparse Attention on Chain-of-thought tokEnization), a novel generative framework for multi-behavior sequential recommendation. GRACE introduces a hybrid Chain-of-Thought (CoT) tokenization method that encodes user-item interactions with explicit attributes from product knowledge graphs (e.g., category, brand, price) over semantic tokenization, enabling interpretable and behavior-aligned generation. To address the inefficiency of standard attention, we design a Journey-Aware Sparse Attention (JSA) mechanism, which selectively attends to compressed, intra-, inter-, and current-context segments in the tokenized sequence. Experiments on two real-world datasets show that GRACE significantly outperforms state-of-the-art baselines, achieving up to +106.9% HR@10 and +106.7% NDCG@10 improvement over the state-of-the-art baseline on the Home domain, and +22.1% HR@10 on the Electronics domain. GRACE also reduces attention computation by up to 48% with long sequences. |
| title | GRACE: Generative Recommendation via Journey-Aware Sparse Attention on Chain-of-Thought Tokenization |
| topic | Computation and Language Artificial Intelligence Information Retrieval |
| url | https://arxiv.org/abs/2507.14758 |