All that is English may be Hindi: Enhancing language identification through automatic ranking of likeliness of word borrowing in social media
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2017
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916269825982464 |
|---|---|
| author | Patro, Jasabanta Samanta, Bidisha Singh, Saurabh Basu, Abhipsa Mukherjee, Prithwish Choudhury, Monojit Mukherjee, Animesh |
| author_facet | Patro, Jasabanta Samanta, Bidisha Singh, Saurabh Basu, Abhipsa Mukherjee, Prithwish Choudhury, Monojit Mukherjee, Animesh |
| contents | In this paper, we present a set of computational methods to identify the likeliness of a word being borrowed, based on the signals from social media. In terms of Spearman correlation coefficient values, our methods perform more than two times better (nearly 0.62) in predicting the borrowing likeliness compared to the best performing baseline (nearly 0.26) reported in literature. Based on this likeliness estimate we asked annotators to re-annotate the language tags of foreign words in predominantly native contexts. In 88 percent of cases the annotators felt that the foreign language tag should be replaced by native language tag, thus indicating a huge scope for improvement of automatic language identification systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_1707_08446 |
| institution | arXiv |
| publishDate | 2017 |
| record_format | arxiv |
| spellingShingle | All that is English may be Hindi: Enhancing language identification through automatic ranking of likeliness of word borrowing in social media Patro, Jasabanta Samanta, Bidisha Singh, Saurabh Basu, Abhipsa Mukherjee, Prithwish Choudhury, Monojit Mukherjee, Animesh Computation and Language In this paper, we present a set of computational methods to identify the likeliness of a word being borrowed, based on the signals from social media. In terms of Spearman correlation coefficient values, our methods perform more than two times better (nearly 0.62) in predicting the borrowing likeliness compared to the best performing baseline (nearly 0.26) reported in literature. Based on this likeliness estimate we asked annotators to re-annotate the language tags of foreign words in predominantly native contexts. In 88 percent of cases the annotators felt that the foreign language tag should be replaced by native language tag, thus indicating a huge scope for improvement of automatic language identification systems. |
| title | All that is English may be Hindi: Enhancing language identification through automatic ranking of likeliness of word borrowing in social media |
| topic | Computation and Language |
| url | https://arxiv.org/abs/1707.08446 |