Towards Red Teaming in Multimodal and Multilingual Translation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911766811770880 |
|---|---|
| author | Ropers, Christophe Dale, David Hansanti, Prangthip Gonzalez, Gabriel Mejia Evtimov, Ivan Wong, Corinne Touret, Christophe Pereyra, Kristina Kim, Seohyun Sonia Ferrer, Cristian Canton Andrews, Pierre Costa-jussà, Marta R. |
| author_facet | Ropers, Christophe Dale, David Hansanti, Prangthip Gonzalez, Gabriel Mejia Evtimov, Ivan Wong, Corinne Touret, Christophe Pereyra, Kristina Kim, Seohyun Sonia Ferrer, Cristian Canton Andrews, Pierre Costa-jussà, Marta R. |
| contents | Assessing performance in Natural Language Processing is becoming increasingly complex. One particular challenge is the potential for evaluation datasets to overlap with training data, either directly or indirectly, which can lead to skewed results and overestimation of model performance. As a consequence, human evaluation is gaining increasing interest as a means to assess the performance and reliability of models. One such method is the red teaming approach, which aims to generate edge cases where a model will produce critical errors. While this methodology is becoming standard practice for generative AI, its application to the realm of conditional AI remains largely unexplored. This paper presents the first study on human-based red teaming for Machine Translation (MT), marking a significant step towards understanding and improving the performance of translation models. We delve into both human-based red teaming and a study on automation, reporting lessons learned and providing recommendations for both translation models and red teaming drills. This pioneering work opens up new avenues for research and development in the field of MT. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_16247 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Towards Red Teaming in Multimodal and Multilingual Translation Ropers, Christophe Dale, David Hansanti, Prangthip Gonzalez, Gabriel Mejia Evtimov, Ivan Wong, Corinne Touret, Christophe Pereyra, Kristina Kim, Seohyun Sonia Ferrer, Cristian Canton Andrews, Pierre Costa-jussà, Marta R. Computation and Language Computers and Society I.2.7 Assessing performance in Natural Language Processing is becoming increasingly complex. One particular challenge is the potential for evaluation datasets to overlap with training data, either directly or indirectly, which can lead to skewed results and overestimation of model performance. As a consequence, human evaluation is gaining increasing interest as a means to assess the performance and reliability of models. One such method is the red teaming approach, which aims to generate edge cases where a model will produce critical errors. While this methodology is becoming standard practice for generative AI, its application to the realm of conditional AI remains largely unexplored. This paper presents the first study on human-based red teaming for Machine Translation (MT), marking a significant step towards understanding and improving the performance of translation models. We delve into both human-based red teaming and a study on automation, reporting lessons learned and providing recommendations for both translation models and red teaming drills. This pioneering work opens up new avenues for research and development in the field of MT. |
| title | Towards Red Teaming in Multimodal and Multilingual Translation |
| topic | Computation and Language Computers and Society I.2.7 |
| url | https://arxiv.org/abs/2401.16247 |