Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2502.16483 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916626101698560 |
|---|---|
| author | Yang, Zhou Pang, Yucai Yin, Hongbo Xiao, Yunpeng |
| author_facet | Yang, Zhou Pang, Yucai Yin, Hongbo Xiao, Yunpeng |
| contents | This paper introduces a new Transformer, called MS$^2$Dformer, that can be used as a generalized backbone for multi-modal sequence spammer detection. Spammer detection is a complex multi-modal task, thus the challenges of applying Transformer are two-fold. Firstly, complex multi-modal noisy information about users can interfere with feature mining. Secondly, the long sequence of users' historical behaviors also puts a huge GPU memory pressure on the attention computation. To solve these problems, we first design a user behavior Tokenization algorithm based on the multi-modal variational autoencoder (MVAE). Subsequently, a hierarchical split-window multi-head attention (SW/W-MHA) mechanism is proposed. The split-window strategy transforms the ultra-long sequences hierarchically into a combination of intra-window short-term and inter-window overall attention. Pre-trained on the public datasets, MS$^2$Dformer's performance far exceeds the previous state of the art. The experiments demonstrate MS$^2$Dformer's ability to act as a backbone. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_16483 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | A Split-Window Transformer for Multi-Model Sequence Spammer Detection using Multi-Model Variational Autoencoder Yang, Zhou Pang, Yucai Yin, Hongbo Xiao, Yunpeng Machine Learning Artificial Intelligence Multimedia Social and Information Networks This paper introduces a new Transformer, called MS$^2$Dformer, that can be used as a generalized backbone for multi-modal sequence spammer detection. Spammer detection is a complex multi-modal task, thus the challenges of applying Transformer are two-fold. Firstly, complex multi-modal noisy information about users can interfere with feature mining. Secondly, the long sequence of users' historical behaviors also puts a huge GPU memory pressure on the attention computation. To solve these problems, we first design a user behavior Tokenization algorithm based on the multi-modal variational autoencoder (MVAE). Subsequently, a hierarchical split-window multi-head attention (SW/W-MHA) mechanism is proposed. The split-window strategy transforms the ultra-long sequences hierarchically into a combination of intra-window short-term and inter-window overall attention. Pre-trained on the public datasets, MS$^2$Dformer's performance far exceeds the previous state of the art. The experiments demonstrate MS$^2$Dformer's ability to act as a backbone. |
| title | A Split-Window Transformer for Multi-Model Sequence Spammer Detection using Multi-Model Variational Autoencoder |
| topic | Machine Learning Artificial Intelligence Multimedia Social and Information Networks |
| url | https://arxiv.org/abs/2502.16483 |