Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.14626 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912076267520000 |
|---|---|
| author | Baumartz, Daniel Bagci, Mevlüt Henlein, Alexander Konca, Maxim Lücking, Andy Mehler, Alexander |
| author_facet | Baumartz, Daniel Bagci, Mevlüt Henlein, Alexander Konca, Maxim Lücking, Andy Mehler, Alexander |
| contents | If sentiment analysis tools were valid classifiers, one would expect them to provide comparable results for sentiment classification on different kinds of corpora and for different languages. In line with results of previous studies we show that sentiment analysis tools disagree on the same dataset. Going beyond previous studies we show that the sentiment tool used for sentiment annotation can even be predicted from its outcome, revealing an algorithmic bias of sentiment analysis. Based on Twitter, Wikipedia and different news corpora from the English, German and French languages, our classifiers separate sentiment tools with an averaged F1-score of 0.89 (for the English corpora). We therefore warn against taking sentiment annotations as face value and argue for the need of more and systematic NLP evaluation studies. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_14626 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | You Shall Know a Tool by the Traces it Leaves: The Predictability of Sentiment Analysis Tools Baumartz, Daniel Bagci, Mevlüt Henlein, Alexander Konca, Maxim Lücking, Andy Mehler, Alexander Computation and Language If sentiment analysis tools were valid classifiers, one would expect them to provide comparable results for sentiment classification on different kinds of corpora and for different languages. In line with results of previous studies we show that sentiment analysis tools disagree on the same dataset. Going beyond previous studies we show that the sentiment tool used for sentiment annotation can even be predicted from its outcome, revealing an algorithmic bias of sentiment analysis. Based on Twitter, Wikipedia and different news corpora from the English, German and French languages, our classifiers separate sentiment tools with an averaged F1-score of 0.89 (for the English corpora). We therefore warn against taking sentiment annotations as face value and argue for the need of more and systematic NLP evaluation studies. |
| title | You Shall Know a Tool by the Traces it Leaves: The Predictability of Sentiment Analysis Tools |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2410.14626 |