Addressing Information Loss and Interaction Collapse: A Dual Enhanced Attention Framework for Feature Interaction
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909537267613696 |
|---|---|
| author | Xu, Yi Lu, Zhiyuan Li, Xiaochen Hu, Jinxin Wen, Hong Chen, Zulong Zhang, Yu Zhang, Jing |
| author_facet | Xu, Yi Lu, Zhiyuan Li, Xiaochen Hu, Jinxin Wen, Hong Chen, Zulong Zhang, Yu Zhang, Jing |
| contents | The Transformer has proven to be a significant approach in feature interaction for CTR prediction, achieving considerable success in previous works. However, it also presents potential challenges in handling feature interactions. Firstly, Transformers may encounter information loss when capturing feature interactions. By relying on inner products to represent pairwise relationships, they compress raw interaction information, which can result in a degradation of fidelity. Secondly, due to the long-tail features distribution, feature fields with low information-abundance embeddings constrain the information abundance of other fields, leading to collapsed embedding matrices. To tackle these issues, we propose a Dual Attention Framework for Enhanced Feature Interaction, known as Dual Enhanced Attention. This framework integrates two attention mechanisms: the Combo-ID attention mechanism and the collapse-avoiding attention mechanism. The Combo-ID attention mechanism directly retains feature interaction pairs to mitigate information loss, while the collapse-avoiding attention mechanism adaptively filters out low information-abundance interaction pairs to prevent interaction collapse. Extensive experiments conducted on industrial datasets have shown the effectiveness of Dual Enhanced Attention. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_11233 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Addressing Information Loss and Interaction Collapse: A Dual Enhanced Attention Framework for Feature Interaction Xu, Yi Lu, Zhiyuan Li, Xiaochen Hu, Jinxin Wen, Hong Chen, Zulong Zhang, Yu Zhang, Jing Information Retrieval Machine Learning The Transformer has proven to be a significant approach in feature interaction for CTR prediction, achieving considerable success in previous works. However, it also presents potential challenges in handling feature interactions. Firstly, Transformers may encounter information loss when capturing feature interactions. By relying on inner products to represent pairwise relationships, they compress raw interaction information, which can result in a degradation of fidelity. Secondly, due to the long-tail features distribution, feature fields with low information-abundance embeddings constrain the information abundance of other fields, leading to collapsed embedding matrices. To tackle these issues, we propose a Dual Attention Framework for Enhanced Feature Interaction, known as Dual Enhanced Attention. This framework integrates two attention mechanisms: the Combo-ID attention mechanism and the collapse-avoiding attention mechanism. The Combo-ID attention mechanism directly retains feature interaction pairs to mitigate information loss, while the collapse-avoiding attention mechanism adaptively filters out low information-abundance interaction pairs to prevent interaction collapse. Extensive experiments conducted on industrial datasets have shown the effectiveness of Dual Enhanced Attention. |
| title | Addressing Information Loss and Interaction Collapse: A Dual Enhanced Attention Framework for Feature Interaction |
| topic | Information Retrieval Machine Learning |
| url | https://arxiv.org/abs/2503.11233 |