Evaluating Large Language Model Capability in Vietnamese Fact-Checking Data Generation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909380662788096 |
|---|---|
| author | To, Long Truong Le, Hung Tuan Nguyen, Dat Van-Thanh Nguyen, Manh Trong Nguyen, Tri Thien Van Huynh, Tin Van Nguyen, Kiet |
| author_facet | To, Long Truong Le, Hung Tuan Nguyen, Dat Van-Thanh Nguyen, Manh Trong Nguyen, Tri Thien Van Huynh, Tin Van Nguyen, Kiet |
| contents | Large Language Models (LLMs), with gradually improving reading comprehension and reasoning capabilities, are being applied to a range of complex language tasks, including the automatic generation of language data for various purposes. However, research on applying LLMs for automatic data generation in low-resource languages like Vietnamese is still underdeveloped and lacks comprehensive evaluation. In this paper, we explore the use of LLMs for automatic data generation for the Vietnamese fact-checking task, which faces significant data limitations. Specifically, we focus on fact-checking data where claims are synthesized from multiple evidence sentences to assess the information synthesis capabilities of LLMs. We develop an automatic data construction process using simple prompt techniques on LLMs and explore several methods to improve the quality of the generated data. To evaluate the quality of the data generated by LLMs, we conduct both manual quality assessments and performance evaluations using language models. Experimental results and manual evaluations illustrate that while the quality of the generated data has significantly improved through fine-tuning techniques, LLMs still cannot match the data quality produced by humans. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_05641 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Evaluating Large Language Model Capability in Vietnamese Fact-Checking Data Generation To, Long Truong Le, Hung Tuan Nguyen, Dat Van-Thanh Nguyen, Manh Trong Nguyen, Tri Thien Van Huynh, Tin Van Nguyen, Kiet Computation and Language Large Language Models (LLMs), with gradually improving reading comprehension and reasoning capabilities, are being applied to a range of complex language tasks, including the automatic generation of language data for various purposes. However, research on applying LLMs for automatic data generation in low-resource languages like Vietnamese is still underdeveloped and lacks comprehensive evaluation. In this paper, we explore the use of LLMs for automatic data generation for the Vietnamese fact-checking task, which faces significant data limitations. Specifically, we focus on fact-checking data where claims are synthesized from multiple evidence sentences to assess the information synthesis capabilities of LLMs. We develop an automatic data construction process using simple prompt techniques on LLMs and explore several methods to improve the quality of the generated data. To evaluate the quality of the data generated by LLMs, we conduct both manual quality assessments and performance evaluations using language models. Experimental results and manual evaluations illustrate that while the quality of the generated data has significantly improved through fine-tuning techniques, LLMs still cannot match the data quality produced by humans. |
| title | Evaluating Large Language Model Capability in Vietnamese Fact-Checking Data Generation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2411.05641 |