Beyond Shallow Heuristics: Leveraging Human Intuition for Curriculum Learning
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914009125486592 |
|---|---|
| author | Toborek, Vanessa Müller, Sebastian Selbach, Tim Horváth, Tamás Bauckhage, Christian |
| author_facet | Toborek, Vanessa Müller, Sebastian Selbach, Tim Horváth, Tamás Bauckhage, Christian |
| contents | Curriculum learning (CL) aims to improve training by presenting data from "easy" to "hard", yet defining and measuring linguistic difficulty remains an open challenge. We investigate whether human-curated simple language can serve as an effective signal for CL. Using the article-level labels from the Simple Wikipedia corpus, we compare label-based curricula to competence-based strategies relying on shallow heuristics. Our experiments with a BERT-tiny model show that adding simple data alone yields no clear benefit. However, structuring it via a curriculum -- especially when introduced first -- consistently improves perplexity, particularly on simple language. In contrast, competence-based curricula lead to no consistent gains over random ordering, probably because they fail to effectively separate the two classes. Our results suggest that human intuition about linguistic difficulty can guide CL for language model pre-training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_19873 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Beyond Shallow Heuristics: Leveraging Human Intuition for Curriculum Learning Toborek, Vanessa Müller, Sebastian Selbach, Tim Horváth, Tamás Bauckhage, Christian Computation and Language Curriculum learning (CL) aims to improve training by presenting data from "easy" to "hard", yet defining and measuring linguistic difficulty remains an open challenge. We investigate whether human-curated simple language can serve as an effective signal for CL. Using the article-level labels from the Simple Wikipedia corpus, we compare label-based curricula to competence-based strategies relying on shallow heuristics. Our experiments with a BERT-tiny model show that adding simple data alone yields no clear benefit. However, structuring it via a curriculum -- especially when introduced first -- consistently improves perplexity, particularly on simple language. In contrast, competence-based curricula lead to no consistent gains over random ordering, probably because they fail to effectively separate the two classes. Our results suggest that human intuition about linguistic difficulty can guide CL for language model pre-training. |
| title | Beyond Shallow Heuristics: Leveraging Human Intuition for Curriculum Learning |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2508.19873 |