Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2310.05696 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929641475801088 |
|---|---|
| author | Abourayya, Amr Kleesiek, Jens Rao, Kanishka Ayday, Erman Rao, Bharat Webb, Geoff Kamp, Michael |
| author_facet | Abourayya, Amr Kleesiek, Jens Rao, Kanishka Ayday, Erman Rao, Bharat Webb, Geoff Kamp, Michael |
| contents | In many critical applications, sensitive data is inherently distributed and cannot be centralized due to privacy concerns. A wide range of federated learning approaches have been proposed to train models locally at each client without sharing their sensitive data, typically by exchanging model parameters, or probabilistic predictions (soft labels) on a public dataset or a combination of both. However, these methods still disclose private information and restrict local models to those that can be trained using gradient-based methods. We propose a federated co-training (FedCT) approach that improves privacy by sharing only definitive (hard) labels on a public unlabeled dataset. Clients use a consensus of these shared labels as pseudo-labels for local training. This federated co-training approach empirically enhances privacy without compromising model quality. In addition, it allows the use of local models that are not suitable for parameter aggregation in traditional federated learning, such as gradient-boosted decision trees, rule ensembles, and random forests. Furthermore, we observe that FedCT performs effectively in federated fine-tuning of large language models, where its pseudo-labeling mechanism is particularly beneficial. Empirical evaluations and theoretical analyses suggest its applicability across a range of federated learning scenarios. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_05696 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Little is Enough: Boosting Privacy by Sharing Only Hard Labels in Federated Semi-Supervised Learning Abourayya, Amr Kleesiek, Jens Rao, Kanishka Ayday, Erman Rao, Bharat Webb, Geoff Kamp, Michael Machine Learning In many critical applications, sensitive data is inherently distributed and cannot be centralized due to privacy concerns. A wide range of federated learning approaches have been proposed to train models locally at each client without sharing their sensitive data, typically by exchanging model parameters, or probabilistic predictions (soft labels) on a public dataset or a combination of both. However, these methods still disclose private information and restrict local models to those that can be trained using gradient-based methods. We propose a federated co-training (FedCT) approach that improves privacy by sharing only definitive (hard) labels on a public unlabeled dataset. Clients use a consensus of these shared labels as pseudo-labels for local training. This federated co-training approach empirically enhances privacy without compromising model quality. In addition, it allows the use of local models that are not suitable for parameter aggregation in traditional federated learning, such as gradient-boosted decision trees, rule ensembles, and random forests. Furthermore, we observe that FedCT performs effectively in federated fine-tuning of large language models, where its pseudo-labeling mechanism is particularly beneficial. Empirical evaluations and theoretical analyses suggest its applicability across a range of federated learning scenarios. |
| title | Little is Enough: Boosting Privacy by Sharing Only Hard Labels in Federated Semi-Supervised Learning |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2310.05696 |