Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2502.00208 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929693513482240 |
|---|---|
| author | Granados, Ana Koroutchev, Kostadin Rodríguez, Francisco de Borja |
| author_facet | Granados, Ana Koroutchev, Kostadin Rodríguez, Francisco de Borja |
| contents | Text datasets can be represented using models that do not preserve text structure, or using models that preserve text structure. Our hypothesis is that depending on the dataset nature, there can be advantages using a model that preserves text structure over one that does not, and viceversa. The key is to determine the best way of representing a particular dataset, based on the dataset itself. In this work, we propose to investigate this problem by combining text distortion and algorithmic clustering based on string compression. Specifically, a distortion technique previously developed by the authors is applied to destroy text structure progressively. Following this, a clustering algorithm based on string compression is used to analyze the effects of the distortion on the information contained in the texts. Several experiments are carried out on text datasets and artificially-generated datasets. The results show that in strongly structural datasets the clustering results worsen as text structure is progressively destroyed. Besides, they show that using a compressor which enables the choice of the size of the left-context symbols helps to determine the nature of the datasets. Finally, the results are contrasted with a method based on multidimensional projections and analogous conclusions are obtained. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_00208 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Discovering Dataset Nature through Algorithmic Clustering based on String Compression Granados, Ana Koroutchev, Kostadin Rodríguez, Francisco de Borja Information Theory Text datasets can be represented using models that do not preserve text structure, or using models that preserve text structure. Our hypothesis is that depending on the dataset nature, there can be advantages using a model that preserves text structure over one that does not, and viceversa. The key is to determine the best way of representing a particular dataset, based on the dataset itself. In this work, we propose to investigate this problem by combining text distortion and algorithmic clustering based on string compression. Specifically, a distortion technique previously developed by the authors is applied to destroy text structure progressively. Following this, a clustering algorithm based on string compression is used to analyze the effects of the distortion on the information contained in the texts. Several experiments are carried out on text datasets and artificially-generated datasets. The results show that in strongly structural datasets the clustering results worsen as text structure is progressively destroyed. Besides, they show that using a compressor which enables the choice of the size of the left-context symbols helps to determine the nature of the datasets. Finally, the results are contrasted with a method based on multidimensional projections and analogous conclusions are obtained. |
| title | Discovering Dataset Nature through Algorithmic Clustering based on String Compression |
| topic | Information Theory |
| url | https://arxiv.org/abs/2502.00208 |