Thai Financial Domain Adaptation of THaLLE -- Technical Report
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915037182951424 |
|---|---|
| author | Labs, KBTG Petchsod, Atthakorn Balee, Pornchanan Khamnuansin, Danupat Lertpiya, Anuruth Saetia, Chanatip Chalothorn, Tawunrat Pongthawornkamol, Thadpong Lertsutthiwong, Monchai |
| author_facet | Labs, KBTG Petchsod, Atthakorn Balee, Pornchanan Khamnuansin, Danupat Lertpiya, Anuruth Saetia, Chanatip Chalothorn, Tawunrat Pongthawornkamol, Thadpong Lertsutthiwong, Monchai |
| contents | Large Language Models (LLMs) excel in general tasks but struggle with domain-specific challenges, such as specialized terminology and localized regulations. Existing financial LLMs, like FinGPT and BloombergGPT, lack support for the Thai financial domain. We developed a Thai Financial LLM using the Investment Consultant (IC) exam dataset from the Stock Exchange of Thailand. To address dataset limitations, we applied data augmentation, ReLoRA for efficient training, Continued Pretraining (CPT) for domain knowledge, and Rank-Stabilized LoRA (rsLoRA) for fine-tuning. Supervised Fine-Tuning (SFT) simulated exam scenarios, while Direct Preference Optimization (DPO) refined the model using feedback. The model achieved scores of 72%, 72%, and 84% on IC exam levels P1, P2, and P3, respectively, demonstrating its effectiveness in Thai financial advisory tasks and its potential for specialized applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_18242 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Thai Financial Domain Adaptation of THaLLE -- Technical Report Labs, KBTG Petchsod, Atthakorn Balee, Pornchanan Khamnuansin, Danupat Lertpiya, Anuruth Saetia, Chanatip Chalothorn, Tawunrat Pongthawornkamol, Thadpong Lertsutthiwong, Monchai Computation and Language Artificial Intelligence Large Language Models (LLMs) excel in general tasks but struggle with domain-specific challenges, such as specialized terminology and localized regulations. Existing financial LLMs, like FinGPT and BloombergGPT, lack support for the Thai financial domain. We developed a Thai Financial LLM using the Investment Consultant (IC) exam dataset from the Stock Exchange of Thailand. To address dataset limitations, we applied data augmentation, ReLoRA for efficient training, Continued Pretraining (CPT) for domain knowledge, and Rank-Stabilized LoRA (rsLoRA) for fine-tuning. Supervised Fine-Tuning (SFT) simulated exam scenarios, while Direct Preference Optimization (DPO) refined the model using feedback. The model achieved scores of 72%, 72%, and 84% on IC exam levels P1, P2, and P3, respectively, demonstrating its effectiveness in Thai financial advisory tasks and its potential for specialized applications. |
| title | Thai Financial Domain Adaptation of THaLLE -- Technical Report |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2411.18242 |