GraSS: Scalable Data Attribution with Gradient Sparsification and Sparse Projection
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908616067383296 |
|---|---|
| author | Hu, Pingbang Melkonian, Joseph Tang, Weijing Zhao, Han Ma, Jiaqi W. |
| author_facet | Hu, Pingbang Melkonian, Joseph Tang, Weijing Zhao, Han Ma, Jiaqi W. |
| contents | Gradient-based data attribution methods, such as influence functions, are critical for understanding the impact of individual training samples without requiring repeated model retraining. However, their scalability is often limited by the high computational and memory costs associated with per-sample gradient computation. In this work, we propose GraSS, a novel gradient compression algorithm and its variants FactGraSS for linear layers specifically, that explicitly leverage the inherent sparsity of per-sample gradients to achieve sub-linear space and time complexity. Extensive experiments demonstrate the effectiveness of our approach, achieving substantial speedups while preserving data influence fidelity. In particular, FactGraSS achieves up to 165% faster throughput on billion-scale models compared to the previous state-of-the-art baselines. Our code is publicly available at https://github.com/TRAIS-Lab/GraSS. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_18976 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | GraSS: Scalable Data Attribution with Gradient Sparsification and Sparse Projection Hu, Pingbang Melkonian, Joseph Tang, Weijing Zhao, Han Ma, Jiaqi W. Machine Learning Artificial Intelligence Gradient-based data attribution methods, such as influence functions, are critical for understanding the impact of individual training samples without requiring repeated model retraining. However, their scalability is often limited by the high computational and memory costs associated with per-sample gradient computation. In this work, we propose GraSS, a novel gradient compression algorithm and its variants FactGraSS for linear layers specifically, that explicitly leverage the inherent sparsity of per-sample gradients to achieve sub-linear space and time complexity. Extensive experiments demonstrate the effectiveness of our approach, achieving substantial speedups while preserving data influence fidelity. In particular, FactGraSS achieves up to 165% faster throughput on billion-scale models compared to the previous state-of-the-art baselines. Our code is publicly available at https://github.com/TRAIS-Lab/GraSS. |
| title | GraSS: Scalable Data Attribution with Gradient Sparsification and Sparse Projection |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2505.18976 |