WangchanLion and WangchanX MRC Eval
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909178643087360 |
|---|---|
| author | Phatthiyaphaibun, Wannaphong Nonesung, Surapon Payoungkhamdee, Patomporn Limkonchotiwat, Peerat Udomcharoenchaikit, Can Sawatphol, Jitkapat Chaksangchaichot, Chompakorn Chuangsuwanich, Ekapol Nutanong, Sarana |
| author_facet | Phatthiyaphaibun, Wannaphong Nonesung, Surapon Payoungkhamdee, Patomporn Limkonchotiwat, Peerat Udomcharoenchaikit, Can Sawatphol, Jitkapat Chaksangchaichot, Chompakorn Chuangsuwanich, Ekapol Nutanong, Sarana |
| contents | This technical report describes the development of WangchanLion, an instruction fine-tuned model focusing on Machine Reading Comprehension (MRC) in the Thai language. Our model is based on SEA-LION and a collection of instruction following datasets. To promote open research and reproducibility, we publicly release all training data, code, and the final model weights under the Apache-2 license. To assess the contextual understanding capability, we conducted extensive experimental studies using two Thai MRC datasets, XQuAD and Iapp_wiki_qa_squad. Experimental results demonstrate the model's ability to comprehend the context and produce an answer faithful to the reference one in 0-shot and 1-shot settings. In addition, our evaluation goes beyond the traditional MRC. We propose a new evaluation scheme assessing the answer's correctness, helpfulness, conciseness, and contextuality. Our code is available publicly at https://github.com/vistec-AI/WangchanLion. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_16127 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | WangchanLion and WangchanX MRC Eval Phatthiyaphaibun, Wannaphong Nonesung, Surapon Payoungkhamdee, Patomporn Limkonchotiwat, Peerat Udomcharoenchaikit, Can Sawatphol, Jitkapat Chaksangchaichot, Chompakorn Chuangsuwanich, Ekapol Nutanong, Sarana Computation and Language Artificial Intelligence This technical report describes the development of WangchanLion, an instruction fine-tuned model focusing on Machine Reading Comprehension (MRC) in the Thai language. Our model is based on SEA-LION and a collection of instruction following datasets. To promote open research and reproducibility, we publicly release all training data, code, and the final model weights under the Apache-2 license. To assess the contextual understanding capability, we conducted extensive experimental studies using two Thai MRC datasets, XQuAD and Iapp_wiki_qa_squad. Experimental results demonstrate the model's ability to comprehend the context and produce an answer faithful to the reference one in 0-shot and 1-shot settings. In addition, our evaluation goes beyond the traditional MRC. We propose a new evaluation scheme assessing the answer's correctness, helpfulness, conciseness, and contextuality. Our code is available publicly at https://github.com/vistec-AI/WangchanLion. |
| title | WangchanLion and WangchanX MRC Eval |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2403.16127 |