TokenFormer: Unify the Multi-Field and Sequential Recommendation Worlds
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914475498536960 |
|---|---|
| author | Zhou, Yifeng Hu, Yuehong Feng, Zhixiang Pan, Junwei Wu, Kaihui Li, Hanyong Zhang, Shangyu Huang, Shudong Zhu, Zhangbin Yin, Chengguo Gu, Haijie Jiang, Jie |
| author_facet | Zhou, Yifeng Hu, Yuehong Feng, Zhixiang Pan, Junwei Wu, Kaihui Li, Hanyong Zhang, Shangyu Huang, Shudong Zhu, Zhangbin Yin, Chengguo Gu, Haijie Jiang, Jie |
| contents | Recommender systems have historically developed along two largely independent paradigms: feature interaction models for modeling correlations among multi-field categorical features, and sequential models for capturing user behavior dynamics from historical interaction sequences. Although recent trends attempt to bridge these paradigms within shared backbones, we empirically reveal that naive unifying these two branches may lead to a failure mode of Sequential Collapse Propagation (SCP). That is, the interaction with those dimensionally ill non-sequence fields leads to the dimensional collapse of the sequence features. To overcome this challenge, we propose TokenFormer, a unified recommendation architecture with the following innovations. First, we introduce a Bottom-Full-Top-Sliding (BFTS) attention scheme, which applies full self-attention in the lower layers and shrinking-window sliding attention in the upper layers. Second, we introduce a Non-Linear Interaction Representation (NLIR) that applies one-sided non-linear multiplicative transformations to the hidden states. Extensive experiments on public benchmarks and Tencent's advertising platform demonstrate state-of-the-art performance, while detailed analysis confirm that TokenFormer significantly improves dimensional robustness and representation discriminability under unified modeling. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_13737 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | TokenFormer: Unify the Multi-Field and Sequential Recommendation Worlds Zhou, Yifeng Hu, Yuehong Feng, Zhixiang Pan, Junwei Wu, Kaihui Li, Hanyong Zhang, Shangyu Huang, Shudong Zhu, Zhangbin Yin, Chengguo Gu, Haijie Jiang, Jie Information Retrieval Artificial Intelligence Recommender systems have historically developed along two largely independent paradigms: feature interaction models for modeling correlations among multi-field categorical features, and sequential models for capturing user behavior dynamics from historical interaction sequences. Although recent trends attempt to bridge these paradigms within shared backbones, we empirically reveal that naive unifying these two branches may lead to a failure mode of Sequential Collapse Propagation (SCP). That is, the interaction with those dimensionally ill non-sequence fields leads to the dimensional collapse of the sequence features. To overcome this challenge, we propose TokenFormer, a unified recommendation architecture with the following innovations. First, we introduce a Bottom-Full-Top-Sliding (BFTS) attention scheme, which applies full self-attention in the lower layers and shrinking-window sliding attention in the upper layers. Second, we introduce a Non-Linear Interaction Representation (NLIR) that applies one-sided non-linear multiplicative transformations to the hidden states. Extensive experiments on public benchmarks and Tencent's advertising platform demonstrate state-of-the-art performance, while detailed analysis confirm that TokenFormer significantly improves dimensional robustness and representation discriminability under unified modeling. |
| title | TokenFormer: Unify the Multi-Field and Sequential Recommendation Worlds |
| topic | Information Retrieval Artificial Intelligence |
| url | https://arxiv.org/abs/2604.13737 |