Experiences from Creating a Benchmark for Sentiment Classification for Varieties of English
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912117540519936 |
|---|---|
| author | Srirag, Dipankar Painter, Jordan Joshi, Aditya Kanojia, Diptesh |
| author_facet | Srirag, Dipankar Painter, Jordan Joshi, Aditya Kanojia, Diptesh |
| contents | Existing benchmarks often fail to account for linguistic diversity, like language variants of English. In this paper, we share our experiences from our ongoing project of building a sentiment classification benchmark for three variants of English: Australian (en-AU), Indian (en-IN), and British (en-UK) English. Using Google Places reviews, we explore the effects of various sampling techniques based on label semantics, review length, and sentiment proportion and report performances on three fine-tuned BERT-based models. Our initial evaluation reveals significant performance variations influenced by sample characteristics, label semantics, and language variety, highlighting the need for nuanced benchmark design. We offer actionable insights for researchers to create robust benchmarks, emphasising the importance of diverse sampling, careful label definition, and comprehensive evaluation across linguistic varieties. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_11216 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Experiences from Creating a Benchmark for Sentiment Classification for Varieties of English Srirag, Dipankar Painter, Jordan Joshi, Aditya Kanojia, Diptesh Computation and Language Existing benchmarks often fail to account for linguistic diversity, like language variants of English. In this paper, we share our experiences from our ongoing project of building a sentiment classification benchmark for three variants of English: Australian (en-AU), Indian (en-IN), and British (en-UK) English. Using Google Places reviews, we explore the effects of various sampling techniques based on label semantics, review length, and sentiment proportion and report performances on three fine-tuned BERT-based models. Our initial evaluation reveals significant performance variations influenced by sample characteristics, label semantics, and language variety, highlighting the need for nuanced benchmark design. We offer actionable insights for researchers to create robust benchmarks, emphasising the importance of diverse sampling, careful label definition, and comprehensive evaluation across linguistic varieties. |
| title | Experiences from Creating a Benchmark for Sentiment Classification for Varieties of English |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2410.11216 |