Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2602.10622 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914321897881600 |
|---|---|
| author | Yuan, Jiahao Xu, Yike Wen, Jinyong Wang, Baokun Chen, Yang Lin, Xiaotong Huang, Wuliang Gao, Ziyi Fu, Xing Cheng, Yu Wang, Weiqiang |
| author_facet | Yuan, Jiahao Xu, Yike Wen, Jinyong Wang, Baokun Chen, Yang Lin, Xiaotong Huang, Wuliang Gao, Ziyi Fu, Xing Cheng, Yu Wang, Weiqiang |
| contents | Decoder-only large language models are increasingly used as behavioral encoders for user representation learning, yet the impact of attention masking on the quality of user embeddings remains underexplored. In this work, we conduct a systematic study of causal, hybrid, and bidirectional attention masks within a unified contrastive learning framework trained on large-scale real-world Alipay data that integrates long-horizon heterogeneous user behaviors. To improve training dynamics when transitioning from causal to bidirectional attention, we propose Gradient-Guided Soft Masking, a gradient-based pre-warmup applied before a linear scheduler that gradually opens future attention during optimization. Evaluated on 9 industrial user cognition benchmarks covering prediction, preference, and marketing sensitivity tasks, our approach consistently yields more stable training and higher-quality bidirectional representations compared with causal, hybrid, and scheduler-only baselines, while remaining compatible with decoder pretraining. Overall, our findings highlight the importance of masking design and training transition in adapting decoder-only LLMs for effective user representation learning. Our code is available at https://github.com/JhCircle/Deepfind-GGSM. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_10622 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | How Do Decoder-Only LLMs Perceive Users? Rethinking Attention Masking for User Representation Learning Yuan, Jiahao Xu, Yike Wen, Jinyong Wang, Baokun Chen, Yang Lin, Xiaotong Huang, Wuliang Gao, Ziyi Fu, Xing Cheng, Yu Wang, Weiqiang Computation and Language Decoder-only large language models are increasingly used as behavioral encoders for user representation learning, yet the impact of attention masking on the quality of user embeddings remains underexplored. In this work, we conduct a systematic study of causal, hybrid, and bidirectional attention masks within a unified contrastive learning framework trained on large-scale real-world Alipay data that integrates long-horizon heterogeneous user behaviors. To improve training dynamics when transitioning from causal to bidirectional attention, we propose Gradient-Guided Soft Masking, a gradient-based pre-warmup applied before a linear scheduler that gradually opens future attention during optimization. Evaluated on 9 industrial user cognition benchmarks covering prediction, preference, and marketing sensitivity tasks, our approach consistently yields more stable training and higher-quality bidirectional representations compared with causal, hybrid, and scheduler-only baselines, while remaining compatible with decoder pretraining. Overall, our findings highlight the importance of masking design and training transition in adapting decoder-only LLMs for effective user representation learning. Our code is available at https://github.com/JhCircle/Deepfind-GGSM. |
| title | How Do Decoder-Only LLMs Perceive Users? Rethinking Attention Masking for User Representation Learning |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2602.10622 |