Do LLMs Find Human Answers To Fact-Driven Questions Perplexing? A Case Study on Reddit
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913293793230848 |
|---|---|
| author | Seegmiller, Parker Gatto, Joseph Sharif, Omar Basak, Madhusudan Preum, Sarah Masud |
| author_facet | Seegmiller, Parker Gatto, Joseph Sharif, Omar Basak, Madhusudan Preum, Sarah Masud |
| contents | Large language models (LLMs) have been shown to be proficient in correctly answering questions in the context of online discourse. However, the study of using LLMs to model human-like answers to fact-driven social media questions is still under-explored. In this work, we investigate how LLMs model the wide variety of human answers to fact-driven questions posed on several topic-specific Reddit communities, or subreddits. We collect and release a dataset of 409 fact-driven questions and 7,534 diverse, human-rated answers from 15 r/Ask{Topic} communities across 3 categories: profession, social identity, and geographic location. We find that LLMs are considerably better at modeling highly-rated human answers to such questions, as opposed to poorly-rated human answers. We present several directions for future research based on our initial findings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_01147 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Do LLMs Find Human Answers To Fact-Driven Questions Perplexing? A Case Study on Reddit Seegmiller, Parker Gatto, Joseph Sharif, Omar Basak, Madhusudan Preum, Sarah Masud Computation and Language Machine Learning Large language models (LLMs) have been shown to be proficient in correctly answering questions in the context of online discourse. However, the study of using LLMs to model human-like answers to fact-driven social media questions is still under-explored. In this work, we investigate how LLMs model the wide variety of human answers to fact-driven questions posed on several topic-specific Reddit communities, or subreddits. We collect and release a dataset of 409 fact-driven questions and 7,534 diverse, human-rated answers from 15 r/Ask{Topic} communities across 3 categories: profession, social identity, and geographic location. We find that LLMs are considerably better at modeling highly-rated human answers to such questions, as opposed to poorly-rated human answers. We present several directions for future research based on our initial findings. |
| title | Do LLMs Find Human Answers To Fact-Driven Questions Perplexing? A Case Study on Reddit |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2404.01147 |