Comprehensive Evaluation of Large Language Models for Topic Modeling
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909231043575808 |
|---|---|
| author | Doi, Tomoki Isonuma, Masaru Yanaka, Hitomi |
| author_facet | Doi, Tomoki Isonuma, Masaru Yanaka, Hitomi |
| contents | Recent work utilizes Large Language Models (LLMs) for topic modeling, generating comprehensible topic labels for given documents. However, their performance has mainly been evaluated qualitatively, and there remains room for quantitative investigation of their capabilities. In this paper, we quantitatively evaluate LLMs from multiple perspectives: the quality of topics, the impact of LLM-specific concerns, such as hallucination and shortcuts for limited documents, and LLMs' controllability of topic categories via prompts. Our findings show that LLMs can identify coherent and diverse topics with few hallucinations but may take shortcuts by focusing only on parts of documents. We also found that their controllability is limited. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_00697 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Comprehensive Evaluation of Large Language Models for Topic Modeling Doi, Tomoki Isonuma, Masaru Yanaka, Hitomi Computation and Language Recent work utilizes Large Language Models (LLMs) for topic modeling, generating comprehensible topic labels for given documents. However, their performance has mainly been evaluated qualitatively, and there remains room for quantitative investigation of their capabilities. In this paper, we quantitatively evaluate LLMs from multiple perspectives: the quality of topics, the impact of LLM-specific concerns, such as hallucination and shortcuts for limited documents, and LLMs' controllability of topic categories via prompts. Our findings show that LLMs can identify coherent and diverse topics with few hallucinations but may take shortcuts by focusing only on parts of documents. We also found that their controllability is limited. |
| title | Comprehensive Evaluation of Large Language Models for Topic Modeling |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2406.00697 |