Laplace Sample Information: Data Informativeness Through a Bayesian Lens
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910959233138688 |
|---|---|
| author | Kaiser, Johannes Schwethelm, Kristian Rueckert, Daniel Kaissis, Georgios |
| author_facet | Kaiser, Johannes Schwethelm, Kristian Rueckert, Daniel Kaissis, Georgios |
| contents | Accurately estimating the informativeness of individual samples in a dataset is an important objective in deep learning, as it can guide sample selection, which can improve model efficiency and accuracy by removing redundant or potentially harmful samples. We propose Laplace Sample Information (LSI) measure of sample informativeness grounded in information theory widely applicable across model architectures and learning settings. LSI leverages a Bayesian approximation to the weight posterior and the KL divergence to measure the change in the parameter distribution induced by a sample of interest from the dataset. We experimentally show that LSI is effective in ordering the data with respect to typicality, detecting mislabeled samples, measuring class-wise informativeness, and assessing dataset difficulty. We demonstrate these capabilities of LSI on image and text data in supervised and unsupervised settings. Moreover, we show that LSI can be computed efficiently through probes and transfers well to the training of large models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_15303 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Laplace Sample Information: Data Informativeness Through a Bayesian Lens Kaiser, Johannes Schwethelm, Kristian Rueckert, Daniel Kaissis, Georgios Machine Learning Artificial Intelligence Information Theory Accurately estimating the informativeness of individual samples in a dataset is an important objective in deep learning, as it can guide sample selection, which can improve model efficiency and accuracy by removing redundant or potentially harmful samples. We propose Laplace Sample Information (LSI) measure of sample informativeness grounded in information theory widely applicable across model architectures and learning settings. LSI leverages a Bayesian approximation to the weight posterior and the KL divergence to measure the change in the parameter distribution induced by a sample of interest from the dataset. We experimentally show that LSI is effective in ordering the data with respect to typicality, detecting mislabeled samples, measuring class-wise informativeness, and assessing dataset difficulty. We demonstrate these capabilities of LSI on image and text data in supervised and unsupervised settings. Moreover, we show that LSI can be computed efficiently through probes and transfers well to the training of large models. |
| title | Laplace Sample Information: Data Informativeness Through a Bayesian Lens |
| topic | Machine Learning Artificial Intelligence Information Theory |
| url | https://arxiv.org/abs/2505.15303 |