Improving the TENOR of Labeling: Re-evaluating Topic Models for Content Analysis
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916131758931968 |
|---|---|
| author | Li, Zongxia Mao, Andrew Stephens, Daniel Goel, Pranav Walpole, Emily Dima, Alden Fung, Juan Boyd-Graber, Jordan |
| author_facet | Li, Zongxia Mao, Andrew Stephens, Daniel Goel, Pranav Walpole, Emily Dima, Alden Fung, Juan Boyd-Graber, Jordan |
| contents | Topic models are a popular tool for understanding text collections, but their evaluation has been a point of contention. Automated evaluation metrics such as coherence are often used, however, their validity has been questioned for neural topic models (NTMs) and can overlook a models benefits in real world applications. To this end, we conduct the first evaluation of neural, supervised and classical topic models in an interactive task based setting. We combine topic models with a classifier and test their ability to help humans conduct content analysis and document annotation. From simulated, real user and expert pilot studies, the Contextual Neural Topic Model does the best on cluster evaluation metrics and human evaluations; however, LDA is competitive with two other NTMs under our simulated experiment and user study results, contrary to what coherence scores suggest. We show that current automated metrics do not provide a complete picture of topic modeling capabilities, but the right choice of NTMs can be better than classical models on practical task. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_16348 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Improving the TENOR of Labeling: Re-evaluating Topic Models for Content Analysis Li, Zongxia Mao, Andrew Stephens, Daniel Goel, Pranav Walpole, Emily Dima, Alden Fung, Juan Boyd-Graber, Jordan Computation and Language Computers and Society Human-Computer Interaction Topic models are a popular tool for understanding text collections, but their evaluation has been a point of contention. Automated evaluation metrics such as coherence are often used, however, their validity has been questioned for neural topic models (NTMs) and can overlook a models benefits in real world applications. To this end, we conduct the first evaluation of neural, supervised and classical topic models in an interactive task based setting. We combine topic models with a classifier and test their ability to help humans conduct content analysis and document annotation. From simulated, real user and expert pilot studies, the Contextual Neural Topic Model does the best on cluster evaluation metrics and human evaluations; however, LDA is competitive with two other NTMs under our simulated experiment and user study results, contrary to what coherence scores suggest. We show that current automated metrics do not provide a complete picture of topic modeling capabilities, but the right choice of NTMs can be better than classical models on practical task. |
| title | Improving the TENOR of Labeling: Re-evaluating Topic Models for Content Analysis |
| topic | Computation and Language Computers and Society Human-Computer Interaction |
| url | https://arxiv.org/abs/2401.16348 |