Dynaword: From One-shot to Continuously Developed Datasets
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916881950048256 |
|---|---|
| author | Enevoldsen, Kenneth Jensen, Kristian Nørgaard Kostkan, Jan Szabó, Balázs Kardos, Márton Vad, Kirten Heinsen, Johan Núñez, Andrea Blasi Barmina, Gianluca Nielsen, Jacob Larsen, Rasmus Vahlstrup, Peter Dalum, Per Møldrup Elliott, Desmond Galke, Lukas Schneider-Kamp, Peter Nielbo, Kristoffer |
| author_facet | Enevoldsen, Kenneth Jensen, Kristian Nørgaard Kostkan, Jan Szabó, Balázs Kardos, Márton Vad, Kirten Heinsen, Johan Núñez, Andrea Blasi Barmina, Gianluca Nielsen, Jacob Larsen, Rasmus Vahlstrup, Peter Dalum, Per Møldrup Elliott, Desmond Galke, Lukas Schneider-Kamp, Peter Nielbo, Kristoffer |
| contents | Large-scale datasets are foundational for research and development in natural language processing. However, current approaches face three key challenges: (1) reliance on ambiguously licensed sources restricting use, sharing, and derivative works; (2) static dataset releases that prevent community contributions and diminish longevity; and (3) quality assurance processes restricted to publishing teams rather than leveraging community expertise.
To address these limitations, we introduce two contributions: the Dynaword approach and Danish Dynaword. The Dynaword approach is a framework for creating large-scale, open datasets that can be continuously updated through community collaboration. Danish Dynaword is a concrete implementation that validates this approach and demonstrates its potential. Danish Dynaword contains over four times as many tokens as comparable releases, is exclusively openly licensed, and has received multiple contributions across industry and research. The repository includes light-weight tests to ensure data formatting, quality, and documentation, establishing a sustainable framework for ongoing community contributions and dataset evolution. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_02271 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Dynaword: From One-shot to Continuously Developed Datasets Enevoldsen, Kenneth Jensen, Kristian Nørgaard Kostkan, Jan Szabó, Balázs Kardos, Márton Vad, Kirten Heinsen, Johan Núñez, Andrea Blasi Barmina, Gianluca Nielsen, Jacob Larsen, Rasmus Vahlstrup, Peter Dalum, Per Møldrup Elliott, Desmond Galke, Lukas Schneider-Kamp, Peter Nielbo, Kristoffer Computation and Language Artificial Intelligence Large-scale datasets are foundational for research and development in natural language processing. However, current approaches face three key challenges: (1) reliance on ambiguously licensed sources restricting use, sharing, and derivative works; (2) static dataset releases that prevent community contributions and diminish longevity; and (3) quality assurance processes restricted to publishing teams rather than leveraging community expertise. To address these limitations, we introduce two contributions: the Dynaword approach and Danish Dynaword. The Dynaword approach is a framework for creating large-scale, open datasets that can be continuously updated through community collaboration. Danish Dynaword is a concrete implementation that validates this approach and demonstrates its potential. Danish Dynaword contains over four times as many tokens as comparable releases, is exclusively openly licensed, and has received multiple contributions across industry and research. The repository includes light-weight tests to ensure data formatting, quality, and documentation, establishing a sustainable framework for ongoing community contributions and dataset evolution. |
| title | Dynaword: From One-shot to Continuously Developed Datasets |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2508.02271 |