CzechTopic: A Benchmark for Zero-Shot Topic Localization in Historical Czech Documents
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915833089884160 |
|---|---|
| author | Kostelník, Martin Hradiš, Michal Dočekal, Martin |
| author_facet | Kostelník, Martin Hradiš, Michal Dočekal, Martin |
| contents | Topic localization aims to identify spans of text that express a given topic defined by a name and description. To study this task, we introduce a human-annotated benchmark based on Czech historical documents, containing human-defined topics together with manually annotated spans and supporting evaluation at both document and word levels. Evaluation is performed relative to human agreement rather than a single reference annotation. We evaluate a diverse range of large language models alongside BERT-based models fine-tuned on a distilled development dataset. Results reveal substantial variability among LLMs, with performance ranging from near-human topic detection to pronounced failures in span localization. While the strongest models approach human agreement, the distilled token embedding models remain competitive despite their smaller scale. The dataset and evaluation framework are publicly available at: https://github.com/dcgm/czechtopic. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_03884 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | CzechTopic: A Benchmark for Zero-Shot Topic Localization in Historical Czech Documents Kostelník, Martin Hradiš, Michal Dočekal, Martin Computation and Language Artificial Intelligence Topic localization aims to identify spans of text that express a given topic defined by a name and description. To study this task, we introduce a human-annotated benchmark based on Czech historical documents, containing human-defined topics together with manually annotated spans and supporting evaluation at both document and word levels. Evaluation is performed relative to human agreement rather than a single reference annotation. We evaluate a diverse range of large language models alongside BERT-based models fine-tuned on a distilled development dataset. Results reveal substantial variability among LLMs, with performance ranging from near-human topic detection to pronounced failures in span localization. While the strongest models approach human agreement, the distilled token embedding models remain competitive despite their smaller scale. The dataset and evaluation framework are publicly available at: https://github.com/dcgm/czechtopic. |
| title | CzechTopic: A Benchmark for Zero-Shot Topic Localization in Historical Czech Documents |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2603.03884 |