Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2504.03541 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908301554352128 |
|---|---|
| author | Kothyari, Mayank Sarawagi, Sunita Chakrabarti, Soumen Arora, Gaurav Merugu, Srujana |
| author_facet | Kothyari, Mayank Sarawagi, Sunita Chakrabarti, Soumen Arora, Gaurav Merugu, Srujana |
| contents | LLMs are increasingly used as seq2seq translators from natural language utterances to structured programs, a process called semantic interpretation. Unlike atomic labels or token sequences, programs are naturally represented as abstract syntax trees (ASTs). Such structured representation raises novel issues related to the design and selection of in-context examples (ICEs) presented to the LLM. We focus on decomposing the pool of available ICE trees into fragments, some of which may be better suited to solving the test instance. Next, we propose how to use (additional invocations of) an LLM with prompted syntax constraints to automatically map the fragments to corresponding utterances. Finally, we adapt and extend a recent method for diverse ICE selection to work with whole and fragmented ICE instances. We evaluate our system, SCUD4ICL, on popular diverse semantic parsing benchmarks, showing visible accuracy gains from our proposed decomposed diverse demonstration method. Benefits are particularly notable for smaller LLMs, ICE pools having larger labeled trees, and programs in lower resource languages. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_03541 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Diverse In-Context Example Selection After Decomposing Programs and Aligned Utterances Improves Semantic Parsing Kothyari, Mayank Sarawagi, Sunita Chakrabarti, Soumen Arora, Gaurav Merugu, Srujana Computation and Language LLMs are increasingly used as seq2seq translators from natural language utterances to structured programs, a process called semantic interpretation. Unlike atomic labels or token sequences, programs are naturally represented as abstract syntax trees (ASTs). Such structured representation raises novel issues related to the design and selection of in-context examples (ICEs) presented to the LLM. We focus on decomposing the pool of available ICE trees into fragments, some of which may be better suited to solving the test instance. Next, we propose how to use (additional invocations of) an LLM with prompted syntax constraints to automatically map the fragments to corresponding utterances. Finally, we adapt and extend a recent method for diverse ICE selection to work with whole and fragmented ICE instances. We evaluate our system, SCUD4ICL, on popular diverse semantic parsing benchmarks, showing visible accuracy gains from our proposed decomposed diverse demonstration method. Benefits are particularly notable for smaller LLMs, ICE pools having larger labeled trees, and programs in lower resource languages. |
| title | Diverse In-Context Example Selection After Decomposing Programs and Aligned Utterances Improves Semantic Parsing |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2504.03541 |