SmurfCat at PAN 2024 TextDetox: Alignment of Multilingual Transformers for Text Detoxification
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917717423947776 |
|---|---|
| author | Rykov, Elisei Zaytsev, Konstantin Anisimov, Ivan Voronin, Alexandr |
| author_facet | Rykov, Elisei Zaytsev, Konstantin Anisimov, Ivan Voronin, Alexandr |
| contents | This paper presents a solution for the Multilingual Text Detoxification task in the PAN-2024 competition of the SmurfCat team. Using data augmentation through machine translation and a special filtering procedure, we collected an additional multilingual parallel dataset for text detoxification. Using the obtained data, we fine-tuned several multilingual sequence-to-sequence models, such as mT0 and Aya, on a text detoxification task. We applied the ORPO alignment technique to the final model. Our final model has only 3.7 billion parameters and achieves state-of-the-art results for the Ukrainian language and near state-of-the-art results for other languages. In the competition, our team achieved first place in the automated evaluation with a score of 0.52 and second place in the final human evaluation with a score of 0.74. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_05449 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SmurfCat at PAN 2024 TextDetox: Alignment of Multilingual Transformers for Text Detoxification Rykov, Elisei Zaytsev, Konstantin Anisimov, Ivan Voronin, Alexandr Computation and Language Artificial Intelligence This paper presents a solution for the Multilingual Text Detoxification task in the PAN-2024 competition of the SmurfCat team. Using data augmentation through machine translation and a special filtering procedure, we collected an additional multilingual parallel dataset for text detoxification. Using the obtained data, we fine-tuned several multilingual sequence-to-sequence models, such as mT0 and Aya, on a text detoxification task. We applied the ORPO alignment technique to the final model. Our final model has only 3.7 billion parameters and achieves state-of-the-art results for the Ukrainian language and near state-of-the-art results for other languages. In the competition, our team achieved first place in the automated evaluation with a score of 0.52 and second place in the final human evaluation with a score of 0.74. |
| title | SmurfCat at PAN 2024 TextDetox: Alignment of Multilingual Transformers for Text Detoxification |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2407.05449 |