An Interpretable and Crosslingual Method for Evaluating Second-Language Dialogues
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916619874205696 |
|---|---|
| author | Gao, Rena Wu, Jingxuan Wu, Xuetong Roever, Carsten Wu, Jing Lv, Long Lau, Jey Han |
| author_facet | Gao, Rena Wu, Jingxuan Wu, Xuetong Roever, Carsten Wu, Jing Lv, Long Lau, Jey Han |
| contents | We analyse the cross-lingual transferability of a dialogue evaluation framework that assesses the relationships between micro-level linguistic features (e.g. backchannels) and macro-level interactivity labels (e.g. topic management), originally designed for English-as-a-second-language dialogues. To this end, we develop CNIMA (Chinese Non-Native Interactivity Measurement and Automation), a Chinese-as-a-second-language labelled dataset with 10K dialogues. We found the evaluation framework to be robust across distinct languages: English and Chinese, revealing language-specific and language-universal relationships between micro-level and macro-level features. Next, we propose an automated, interpretable approach with low data requirement that scores the overall quality of a second-language dialogue based on the framework. Our approach is interpretable in that it reveals the key linguistic and interactivity features that contributed to the overall quality score. As our approach does not require labelled data, it can also be adapted to other languages for second-language dialogue evaluation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_16518 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | An Interpretable and Crosslingual Method for Evaluating Second-Language Dialogues Gao, Rena Wu, Jingxuan Wu, Xuetong Roever, Carsten Wu, Jing Lv, Long Lau, Jey Han Computation and Language We analyse the cross-lingual transferability of a dialogue evaluation framework that assesses the relationships between micro-level linguistic features (e.g. backchannels) and macro-level interactivity labels (e.g. topic management), originally designed for English-as-a-second-language dialogues. To this end, we develop CNIMA (Chinese Non-Native Interactivity Measurement and Automation), a Chinese-as-a-second-language labelled dataset with 10K dialogues. We found the evaluation framework to be robust across distinct languages: English and Chinese, revealing language-specific and language-universal relationships between micro-level and macro-level features. Next, we propose an automated, interpretable approach with low data requirement that scores the overall quality of a second-language dialogue based on the framework. Our approach is interpretable in that it reveals the key linguistic and interactivity features that contributed to the overall quality score. As our approach does not require labelled data, it can also be adapted to other languages for second-language dialogue evaluation. |
| title | An Interpretable and Crosslingual Method for Evaluating Second-Language Dialogues |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2408.16518 |