Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2405.01121 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929542358106112 |
|---|---|
| author | Golany, Lotem Galgani, Filippo Mamo, Maya Parasol, Nimrod Vandsburger, Omer Bar, Nadav Dagan, Ido |
| author_facet | Golany, Lotem Galgani, Filippo Mamo, Maya Parasol, Nimrod Vandsburger, Omer Bar, Nadav Dagan, Ido |
| contents | Automating data generation with Large Language Models (LLMs) has become increasingly popular. In this work, we investigate the feasibility and effectiveness of LLM-based data generation in the challenging setting of source-grounded information-seeking dialogs, with response attribution, over long documents. Our source texts consist of long and noisy meeting transcripts, adding to the task complexity. Since automating attribution remains difficult, we propose a semi-automatic approach: dialog queries and responses are generated with LLMs, followed by human verification and identification of attribution spans. Using this approach, we created MISeD -- Meeting Information Seeking Dialogs dataset -- a dataset of information-seeking dialogs focused on meeting transcripts. Models finetuned with MISeD demonstrate superior performance compared to off-the-shelf models, even those of larger size. Finetuning on MISeD gives comparable response generation quality to finetuning on fully manual data, while improving attribution quality and reducing time and effort. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_01121 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Efficient Data Generation for Source-grounded Information-seeking Dialogs: A Use Case for Meeting Transcripts Golany, Lotem Galgani, Filippo Mamo, Maya Parasol, Nimrod Vandsburger, Omer Bar, Nadav Dagan, Ido Computation and Language Artificial Intelligence Automating data generation with Large Language Models (LLMs) has become increasingly popular. In this work, we investigate the feasibility and effectiveness of LLM-based data generation in the challenging setting of source-grounded information-seeking dialogs, with response attribution, over long documents. Our source texts consist of long and noisy meeting transcripts, adding to the task complexity. Since automating attribution remains difficult, we propose a semi-automatic approach: dialog queries and responses are generated with LLMs, followed by human verification and identification of attribution spans. Using this approach, we created MISeD -- Meeting Information Seeking Dialogs dataset -- a dataset of information-seeking dialogs focused on meeting transcripts. Models finetuned with MISeD demonstrate superior performance compared to off-the-shelf models, even those of larger size. Finetuning on MISeD gives comparable response generation quality to finetuning on fully manual data, while improving attribution quality and reducing time and effort. |
| title | Efficient Data Generation for Source-grounded Information-seeking Dialogs: A Use Case for Meeting Transcripts |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2405.01121 |