Improving User Behavior Prediction: Leveraging Annotator Metadata in Supervised Machine Learning Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916762522484736 |
|---|---|
| author | Ng, Lynnette Hui Xian Jaidka, Kokil Tay, Kaiyuan Ahuja, Hansin Chhaya, Niyati |
| author_facet | Ng, Lynnette Hui Xian Jaidka, Kokil Tay, Kaiyuan Ahuja, Hansin Chhaya, Niyati |
| contents | Supervised machine-learning models often underperform in predicting user behaviors from conversational text, hindered by poor crowdsourced label quality and low NLP task accuracy. We introduce the Metadata-Sensitive Weighted-Encoding Ensemble Model (MSWEEM), which integrates annotator meta-features like fatigue and speeding. First, our results show MSWEEM outperforms standard ensembles by 14% on held-out data and 12% on an alternative dataset. Second, we find that incorporating signals of annotator behavior, such as speed and fatigue, significantly boosts model performance. Third, we find that annotators with higher qualifications, such as Master's, deliver more consistent and faster annotations. Given the increasing uncertainty over annotation quality, our experiments show that understanding annotator patterns is crucial for enhancing model accuracy in user behavior prediction. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_21000 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Improving User Behavior Prediction: Leveraging Annotator Metadata in Supervised Machine Learning Models Ng, Lynnette Hui Xian Jaidka, Kokil Tay, Kaiyuan Ahuja, Hansin Chhaya, Niyati Machine Learning Artificial Intelligence Supervised machine-learning models often underperform in predicting user behaviors from conversational text, hindered by poor crowdsourced label quality and low NLP task accuracy. We introduce the Metadata-Sensitive Weighted-Encoding Ensemble Model (MSWEEM), which integrates annotator meta-features like fatigue and speeding. First, our results show MSWEEM outperforms standard ensembles by 14% on held-out data and 12% on an alternative dataset. Second, we find that incorporating signals of annotator behavior, such as speed and fatigue, significantly boosts model performance. Third, we find that annotators with higher qualifications, such as Master's, deliver more consistent and faster annotations. Given the increasing uncertainty over annotation quality, our experiments show that understanding annotator patterns is crucial for enhancing model accuracy in user behavior prediction. |
| title | Improving User Behavior Prediction: Leveraging Annotator Metadata in Supervised Machine Learning Models |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2503.21000 |