Pairing Analogy-Augmented Generation with Procedural Memory for Procedural Q&A
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913914511425536 |
|---|---|
| author | Roth, K Gupta, Rushil Halle, Simon Liu, Bang |
| author_facet | Roth, K Gupta, Rushil Halle, Simon Liu, Bang |
| contents | Large language models struggle to synthesize disparate pieces of information into a coherent plan when approaching a complex procedural task. In this work, we introduce a novel formalism and structure for such procedural knowledge. Based on this formalism, we present a novel procedural knowledge dataset called LCStep, which we created from LangChain tutorials. To leverage this procedural knowledge to solve new tasks, we propose analogy-augmented generation (AAG), which draws inspiration from the human ability to assimilate past experiences to solve unfamiliar problems. AAG uses a custom procedure memory store to retrieve and adapt specialized domain knowledge to answer new procedural tasks. We demonstrate that AAG outperforms few-shot and RAG baselines on LCStep, RecipeNLG, and CHAMP datasets under a pairwise LLM-based evaluation, corroborated by human evaluation in the case of RecipeNLG. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_01344 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Pairing Analogy-Augmented Generation with Procedural Memory for Procedural Q&A Roth, K Gupta, Rushil Halle, Simon Liu, Bang Artificial Intelligence Computation and Language Large language models struggle to synthesize disparate pieces of information into a coherent plan when approaching a complex procedural task. In this work, we introduce a novel formalism and structure for such procedural knowledge. Based on this formalism, we present a novel procedural knowledge dataset called LCStep, which we created from LangChain tutorials. To leverage this procedural knowledge to solve new tasks, we propose analogy-augmented generation (AAG), which draws inspiration from the human ability to assimilate past experiences to solve unfamiliar problems. AAG uses a custom procedure memory store to retrieve and adapt specialized domain knowledge to answer new procedural tasks. We demonstrate that AAG outperforms few-shot and RAG baselines on LCStep, RecipeNLG, and CHAMP datasets under a pairwise LLM-based evaluation, corroborated by human evaluation in the case of RecipeNLG. |
| title | Pairing Analogy-Augmented Generation with Procedural Memory for Procedural Q&A |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2409.01344 |