Fine-Tuning Language Models Using Formal Methods Feedback
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913291526209536 |
|---|---|
| author | Yang, Yunhao Bhatt, Neel P. Ingebrand, Tyler Ward, William Carr, Steven Wang, Zhangyang Topcu, Ufuk |
| author_facet | Yang, Yunhao Bhatt, Neel P. Ingebrand, Tyler Ward, William Carr, Steven Wang, Zhangyang Topcu, Ufuk |
| contents | Although pre-trained language models encode generic knowledge beneficial for planning and control, they may fail to generate appropriate control policies for domain-specific tasks. Existing fine-tuning methods use human feedback to address this limitation, however, sourcing human feedback is labor intensive and costly. We present a fully automated approach to fine-tune pre-trained language models for applications in autonomous systems, bridging the gap between generic knowledge and domain-specific requirements while reducing cost. The method synthesizes automaton-based controllers from pre-trained models guided by natural language task descriptions. These controllers are verifiable against independently provided specifications within a world model, which can be abstract or obtained from a high-fidelity simulator. Controllers with high compliance with the desired specifications receive higher ranks, guiding the iterative fine-tuning process. We provide quantitative evidences, primarily in autonomous driving, to demonstrate the method's effectiveness across multiple tasks. The results indicate an improvement in percentage of specifications satisfied by the controller from 60% to 90%. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_18239 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Fine-Tuning Language Models Using Formal Methods Feedback Yang, Yunhao Bhatt, Neel P. Ingebrand, Tyler Ward, William Carr, Steven Wang, Zhangyang Topcu, Ufuk Artificial Intelligence Computation and Language Formal Languages and Automata Theory Robotics Although pre-trained language models encode generic knowledge beneficial for planning and control, they may fail to generate appropriate control policies for domain-specific tasks. Existing fine-tuning methods use human feedback to address this limitation, however, sourcing human feedback is labor intensive and costly. We present a fully automated approach to fine-tune pre-trained language models for applications in autonomous systems, bridging the gap between generic knowledge and domain-specific requirements while reducing cost. The method synthesizes automaton-based controllers from pre-trained models guided by natural language task descriptions. These controllers are verifiable against independently provided specifications within a world model, which can be abstract or obtained from a high-fidelity simulator. Controllers with high compliance with the desired specifications receive higher ranks, guiding the iterative fine-tuning process. We provide quantitative evidences, primarily in autonomous driving, to demonstrate the method's effectiveness across multiple tasks. The results indicate an improvement in percentage of specifications satisfied by the controller from 60% to 90%. |
| title | Fine-Tuning Language Models Using Formal Methods Feedback |
| topic | Artificial Intelligence Computation and Language Formal Languages and Automata Theory Robotics |
| url | https://arxiv.org/abs/2310.18239 |