Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.12470 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929546487398400 |
|---|---|
| author | Kohlenberg, Leo Horns, Leonard Sadrieh, Frederic Kiele, Nils Clausen, Matthis Ketterer, Konstantin Navasardyan, Avetis Czinczoll, Tamara de Melo, Gerard Herbrich, Ralf |
| author_facet | Kohlenberg, Leo Horns, Leonard Sadrieh, Frederic Kiele, Nils Clausen, Matthis Ketterer, Konstantin Navasardyan, Avetis Czinczoll, Tamara de Melo, Gerard Herbrich, Ralf |
| contents | Annotating large datasets can be challenging. However, crowd-sourcing is often expensive and can lack quality, especially for non-trivial tasks. We propose a method of using LLMs as few-shot learners for annotating data in a complex natural language task where we learn a standalone model to predict usage options for products from customer reviews. We also propose a new evaluation metric for this scenario, HAMS4, that can be used to compare a set of strings with multiple reference sets. Learning a custom model offers individual control over energy efficiency and privacy measures compared to using the LLM directly for the sequence-to-sequence task. We compare this data annotation approach with other traditional methods and demonstrate how LLMs can enable considerable cost savings. We find that the quality of the resulting data exceeds the level attained by third-party vendor services and that GPT-4-generated labels even reach the level of domain experts. We make the code and generated labels publicly available. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_12470 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Learning to Predict Usage Options of Product Reviews with LLM-Generated Labels Kohlenberg, Leo Horns, Leonard Sadrieh, Frederic Kiele, Nils Clausen, Matthis Ketterer, Konstantin Navasardyan, Avetis Czinczoll, Tamara de Melo, Gerard Herbrich, Ralf Computation and Language Annotating large datasets can be challenging. However, crowd-sourcing is often expensive and can lack quality, especially for non-trivial tasks. We propose a method of using LLMs as few-shot learners for annotating data in a complex natural language task where we learn a standalone model to predict usage options for products from customer reviews. We also propose a new evaluation metric for this scenario, HAMS4, that can be used to compare a set of strings with multiple reference sets. Learning a custom model offers individual control over energy efficiency and privacy measures compared to using the LLM directly for the sequence-to-sequence task. We compare this data annotation approach with other traditional methods and demonstrate how LLMs can enable considerable cost savings. We find that the quality of the resulting data exceeds the level attained by third-party vendor services and that GPT-4-generated labels even reach the level of domain experts. We make the code and generated labels publicly available. |
| title | Learning to Predict Usage Options of Product Reviews with LLM-Generated Labels |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2410.12470 |