ProxAnn: Use-Oriented Evaluations of Topic Models and Document Clustering
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908429469089792 |
|---|---|
| author | Hoyle, Alexander Calvo-Bartolomé, Lorena Boyd-Graber, Jordan Resnik, Philip |
| author_facet | Hoyle, Alexander Calvo-Bartolomé, Lorena Boyd-Graber, Jordan Resnik, Philip |
| contents | Topic model and document-clustering evaluations either use automated metrics that align poorly with human preferences or require expert labels that are intractable to scale. We design a scalable human evaluation protocol and a corresponding automated approximation that reflect practitioners' real-world usage of models. Annotators -- or an LLM-based proxy -- review text items assigned to a topic or cluster, infer a category for the group, then apply that category to other documents. Using this protocol, we collect extensive crowdworker annotations of outputs from a diverse set of topic models on two datasets. We then use these annotations to validate automated proxies, finding that the best LLM proxies are statistically indistinguishable from a human annotator and can therefore serve as a reasonable substitute in automated evaluations. Package, web interface, and data are at https://github.com/ahoho/proxann |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_00828 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ProxAnn: Use-Oriented Evaluations of Topic Models and Document Clustering Hoyle, Alexander Calvo-Bartolomé, Lorena Boyd-Graber, Jordan Resnik, Philip Computation and Language Topic model and document-clustering evaluations either use automated metrics that align poorly with human preferences or require expert labels that are intractable to scale. We design a scalable human evaluation protocol and a corresponding automated approximation that reflect practitioners' real-world usage of models. Annotators -- or an LLM-based proxy -- review text items assigned to a topic or cluster, infer a category for the group, then apply that category to other documents. Using this protocol, we collect extensive crowdworker annotations of outputs from a diverse set of topic models on two datasets. We then use these annotations to validate automated proxies, finding that the best LLM proxies are statistically indistinguishable from a human annotator and can therefore serve as a reasonable substitute in automated evaluations. Package, web interface, and data are at https://github.com/ahoho/proxann |
| title | ProxAnn: Use-Oriented Evaluations of Topic Models and Document Clustering |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2507.00828 |