Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2407.00478 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917870639775744 |
|---|---|
| author | Yao, Quanming Zhang, Yongqi Wang, Yaqing Yin, Nan Kwok, James Yang, Qiang |
| author_facet | Yao, Quanming Zhang, Yongqi Wang, Yaqing Yin, Nan Kwok, James Yang, Qiang |
| contents | The brute-force scaleup of training datasets, learnable parameters and computation power, has become a prevalent strategy for developing more robust learning models. However, due to bottlenecks in data, computation, and trust, the sustainability of this strategy is a serious concern. In this paper, we attempt to address this issue in a parsimonious manner (i.e., achieving greater potential with simpler models). The key is to drive models using domain-specific knowledge, such as symbols, logic, and formulas, instead of purely relying on scaleup. This approach allows us to build a framework that uses this knowledge as "building blocks" to achieve parsimony in model design, training, and interpretation. Empirical results show that our methods surpass those that typically follow the scaling law. We also demonstrate our framework in AI for science, specifically in the problem of drug-drug interaction prediction. We hope our research can foster more diverse technical roadmaps in the era of foundation models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_00478 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Beyond Scaleup: Knowledge-aware Parsimony Learning from Deep Networks Yao, Quanming Zhang, Yongqi Wang, Yaqing Yin, Nan Kwok, James Yang, Qiang Machine Learning Artificial Intelligence The brute-force scaleup of training datasets, learnable parameters and computation power, has become a prevalent strategy for developing more robust learning models. However, due to bottlenecks in data, computation, and trust, the sustainability of this strategy is a serious concern. In this paper, we attempt to address this issue in a parsimonious manner (i.e., achieving greater potential with simpler models). The key is to drive models using domain-specific knowledge, such as symbols, logic, and formulas, instead of purely relying on scaleup. This approach allows us to build a framework that uses this knowledge as "building blocks" to achieve parsimony in model design, training, and interpretation. Empirical results show that our methods surpass those that typically follow the scaling law. We also demonstrate our framework in AI for science, specifically in the problem of drug-drug interaction prediction. We hope our research can foster more diverse technical roadmaps in the era of foundation models. |
| title | Beyond Scaleup: Knowledge-aware Parsimony Learning from Deep Networks |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2407.00478 |