QA-TOOLBOX: Conversational Question-Answering for process task guidance in manufacturing
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909414219317248 |
|---|---|
| author | Manuvinakurike, Ramesh Watkins, Elizabeth Savur, Celal Rhodes, Anthony Biswas, Sovan Mejia, Gesem Gudino Beckwith, Richard Sahay, Saurav Raffa, Giuseppe Nachman, Lama |
| author_facet | Manuvinakurike, Ramesh Watkins, Elizabeth Savur, Celal Rhodes, Anthony Biswas, Sovan Mejia, Gesem Gudino Beckwith, Richard Sahay, Saurav Raffa, Giuseppe Nachman, Lama |
| contents | In this work we explore utilizing LLMs for data augmentation for manufacturing task guidance system. The dataset consists of representative samples of interactions with technicians working in an advanced manufacturing setting. The purpose of this work to explore the task, data augmentation for the supported tasks and evaluating the performance of the existing LLMs. We observe that that task is complex requiring understanding from procedure specification documents, actions and objects sequenced temporally. The dataset consists of 200,000+ question/answer pairs that refer to the spec document and are grounded in narrations and/or video demonstrations. We compared the performance of several popular open-sourced LLMs by developing a baseline using each LLM and then compared the responses in a reference-free setting using LLM-as-a-judge and compared the ratings with crowd-workers whilst validating the ratings with experts. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_02638 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | QA-TOOLBOX: Conversational Question-Answering for process task guidance in manufacturing Manuvinakurike, Ramesh Watkins, Elizabeth Savur, Celal Rhodes, Anthony Biswas, Sovan Mejia, Gesem Gudino Beckwith, Richard Sahay, Saurav Raffa, Giuseppe Nachman, Lama Computation and Language Artificial Intelligence In this work we explore utilizing LLMs for data augmentation for manufacturing task guidance system. The dataset consists of representative samples of interactions with technicians working in an advanced manufacturing setting. The purpose of this work to explore the task, data augmentation for the supported tasks and evaluating the performance of the existing LLMs. We observe that that task is complex requiring understanding from procedure specification documents, actions and objects sequenced temporally. The dataset consists of 200,000+ question/answer pairs that refer to the spec document and are grounded in narrations and/or video demonstrations. We compared the performance of several popular open-sourced LLMs by developing a baseline using each LLM and then compared the responses in a reference-free setting using LLM-as-a-judge and compared the ratings with crowd-workers whilst validating the ratings with experts. |
| title | QA-TOOLBOX: Conversational Question-Answering for process task guidance in manufacturing |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2412.02638 |