Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2503.21000 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916762522484736 |
|---|---|
| author | Ng, Lynnette Hui Xian Jaidka, Kokil Tay, Kaiyuan Ahuja, Hansin Chhaya, Niyati |
| author_facet | Ng, Lynnette Hui Xian Jaidka, Kokil Tay, Kaiyuan Ahuja, Hansin Chhaya, Niyati |
| contents | Supervised machine-learning models often underperform in predicting user behaviors from conversational text, hindered by poor crowdsourced label quality and low NLP task accuracy. We introduce the Metadata-Sensitive Weighted-Encoding Ensemble Model (MSWEEM), which integrates annotator meta-features like fatigue and speeding. First, our results show MSWEEM outperforms standard ensembles by 14% on held-out data and 12% on an alternative dataset. Second, we find that incorporating signals of annotator behavior, such as speed and fatigue, significantly boosts model performance. Third, we find that annotators with higher qualifications, such as Master's, deliver more consistent and faster annotations. Given the increasing uncertainty over annotation quality, our experiments show that understanding annotator patterns is crucial for enhancing model accuracy in user behavior prediction. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_21000 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Improving User Behavior Prediction: Leveraging Annotator Metadata in Supervised Machine Learning Models Ng, Lynnette Hui Xian Jaidka, Kokil Tay, Kaiyuan Ahuja, Hansin Chhaya, Niyati Machine Learning Artificial Intelligence Supervised machine-learning models often underperform in predicting user behaviors from conversational text, hindered by poor crowdsourced label quality and low NLP task accuracy. We introduce the Metadata-Sensitive Weighted-Encoding Ensemble Model (MSWEEM), which integrates annotator meta-features like fatigue and speeding. First, our results show MSWEEM outperforms standard ensembles by 14% on held-out data and 12% on an alternative dataset. Second, we find that incorporating signals of annotator behavior, such as speed and fatigue, significantly boosts model performance. Third, we find that annotators with higher qualifications, such as Master's, deliver more consistent and faster annotations. Given the increasing uncertainty over annotation quality, our experiments show that understanding annotator patterns is crucial for enhancing model accuracy in user behavior prediction. |
| title | Improving User Behavior Prediction: Leveraging Annotator Metadata in Supervised Machine Learning Models |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2503.21000 |