An Empirical Study of Validating Synthetic Data for Formula Generation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911051019190272 |
|---|---|
| author | Singh, Usneek Cambronero, José Gulwani, Sumit Kanade, Aditya Khatry, Anirudh Le, Vu Singh, Mukul Verbruggen, Gust |
| author_facet | Singh, Usneek Cambronero, José Gulwani, Sumit Kanade, Aditya Khatry, Anirudh Le, Vu Singh, Mukul Verbruggen, Gust |
| contents | Large language models (LLMs) can be leveraged to help with writing formulas in spreadsheets, but resources on these formulas are scarce, impacting both the base performance of pre-trained models and limiting the ability to fine-tune them. Given a corpus of formulas, we can use a(nother) model to generate synthetic natural language utterances for fine-tuning. However, it is important to validate whether the NL generated by the LLM is indeed accurate to be beneficial for fine-tuning. In this paper, we provide empirical results on the impact of validating these synthetic training examples with surrogate objectives that evaluate the accuracy of the synthetic annotations. We demonstrate that validation improves performance over raw data across four models (2 open and 2 closed weight). Interestingly, we show that although validation tends to prune more challenging examples, it increases the complexity of problems that models can solve after being fine-tuned on validated data. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_10657 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | An Empirical Study of Validating Synthetic Data for Formula Generation Singh, Usneek Cambronero, José Gulwani, Sumit Kanade, Aditya Khatry, Anirudh Le, Vu Singh, Mukul Verbruggen, Gust Computation and Language Artificial Intelligence Large language models (LLMs) can be leveraged to help with writing formulas in spreadsheets, but resources on these formulas are scarce, impacting both the base performance of pre-trained models and limiting the ability to fine-tune them. Given a corpus of formulas, we can use a(nother) model to generate synthetic natural language utterances for fine-tuning. However, it is important to validate whether the NL generated by the LLM is indeed accurate to be beneficial for fine-tuning. In this paper, we provide empirical results on the impact of validating these synthetic training examples with surrogate objectives that evaluate the accuracy of the synthetic annotations. We demonstrate that validation improves performance over raw data across four models (2 open and 2 closed weight). Interestingly, we show that although validation tends to prune more challenging examples, it increases the complexity of problems that models can solve after being fine-tuned on validated data. |
| title | An Empirical Study of Validating Synthetic Data for Formula Generation |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2407.10657 |