Assessing Trustworthiness of AI Training Dataset using Subjective Logic -- A Use Case on Bias
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909032138145792 |
|---|---|
| author | Ouattara, Koffi Ismael Krontiris, Ioannis Dimitrakos, Theo Kargl, Frank |
| author_facet | Ouattara, Koffi Ismael Krontiris, Ioannis Dimitrakos, Theo Kargl, Frank |
| contents | As AI systems increasingly rely on training data, assessing dataset trustworthiness has become critical, particularly for properties like fairness or bias that emerge at the dataset level. Prior work has used Subjective Logic to assess trustworthiness of individual data, but not to evaluate trustworthiness properties that emerge only at the level of the dataset as a whole. This paper introduces the first formal framework for assessing the trustworthiness of AI training datasets, enabling uncertainty-aware evaluations of global properties such as bias. Built on Subjective Logic, our approach supports trust propositions and quantifies uncertainty in scenarios where evidence is incomplete, distributed, and/or conflicting. We instantiate this framework on the trustworthiness property of bias, and we experimentally evaluate it based on a traffic sign recognition dataset. The results demonstrate that our method captures class imbalance and remains interpretable and robust in both centralized and federated contexts. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_13813 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Assessing Trustworthiness of AI Training Dataset using Subjective Logic -- A Use Case on Bias Ouattara, Koffi Ismael Krontiris, Ioannis Dimitrakos, Theo Kargl, Frank Machine Learning Artificial Intelligence As AI systems increasingly rely on training data, assessing dataset trustworthiness has become critical, particularly for properties like fairness or bias that emerge at the dataset level. Prior work has used Subjective Logic to assess trustworthiness of individual data, but not to evaluate trustworthiness properties that emerge only at the level of the dataset as a whole. This paper introduces the first formal framework for assessing the trustworthiness of AI training datasets, enabling uncertainty-aware evaluations of global properties such as bias. Built on Subjective Logic, our approach supports trust propositions and quantifies uncertainty in scenarios where evidence is incomplete, distributed, and/or conflicting. We instantiate this framework on the trustworthiness property of bias, and we experimentally evaluate it based on a traffic sign recognition dataset. The results demonstrate that our method captures class imbalance and remains interpretable and robust in both centralized and federated contexts. |
| title | Assessing Trustworthiness of AI Training Dataset using Subjective Logic -- A Use Case on Bias |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2508.13813 |