JARVIS-Leaderboard: A Large Scale Benchmark of Materials Design Methods
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2023
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909210601586688 |
|---|---|
| author | Choudhary, Kamal Wines, Daniel Li, Kangming Garrity, Kevin F. Gupta, Vishu Romero, Aldo H. Krogel, Jaron T. Saritas, Kayahan Fuhr, Addis Ganesh, Panchapakesan Kent, Paul R. C. Yan, Keqiang Lin, Yuchao Ji, Shuiwang Blaiszik, Ben Reiser, Patrick Friederich, Pascal Agrawal, Ankit Tiwary, Pratyush Beyerle, Eric Minch, Peter Rhone, Trevor David Takeuchi, Ichiro Wexler, Robert B. Mannodi-Kanakkithodi, Arun Ertekin, Elif Mishra, Avanish Mathew, Nithin Baird, Sterling G. Wood, Mitchell Rohskopf, Andrew Dale Hattrick-Simpers, Jason Wang, Shih-Han Achenie, Luke E. K. Xin, Hongliang Williams, Maureen Biacchi, Adam J. Tavazza, Francesca |
| author_facet | Choudhary, Kamal Wines, Daniel Li, Kangming Garrity, Kevin F. Gupta, Vishu Romero, Aldo H. Krogel, Jaron T. Saritas, Kayahan Fuhr, Addis Ganesh, Panchapakesan Kent, Paul R. C. Yan, Keqiang Lin, Yuchao Ji, Shuiwang Blaiszik, Ben Reiser, Patrick Friederich, Pascal Agrawal, Ankit Tiwary, Pratyush Beyerle, Eric Minch, Peter Rhone, Trevor David Takeuchi, Ichiro Wexler, Robert B. Mannodi-Kanakkithodi, Arun Ertekin, Elif Mishra, Avanish Mathew, Nithin Baird, Sterling G. Wood, Mitchell Rohskopf, Andrew Dale Hattrick-Simpers, Jason Wang, Shih-Han Achenie, Luke E. K. Xin, Hongliang Williams, Maureen Biacchi, Adam J. Tavazza, Francesca |
| contents | Lack of rigorous reproducibility and validation are major hurdles for scientific development across many fields. Materials science in particular encompasses a variety of experimental and theoretical approaches that require careful benchmarking. Leaderboard efforts have been developed previously to mitigate these issues. However, a comprehensive comparison and benchmarking on an integrated platform with multiple data modalities with both perfect and defect materials data is still lacking. This work introduces JARVIS-Leaderboard, an open-source and community-driven platform that facilitates benchmarking and enhances reproducibility. The platform allows users to set up benchmarks with custom tasks and enables contributions in the form of dataset, code, and meta-data submissions. We cover the following materials design categories: Artificial Intelligence (AI), Electronic Structure (ES), Force-fields (FF), Quantum Computation (QC) and Experiments (EXP). For AI, we cover several types of input data, including atomic structures, atomistic images, spectra, and text. For ES, we consider multiple ES approaches, software packages, pseudopotentials, materials, and properties, comparing results to experiment. For FF, we compare multiple approaches for material property predictions. For QC, we benchmark Hamiltonian simulations using various quantum algorithms and circuits. Finally, for experiments, we use the inter-laboratory approach to establish benchmarks. There are 1281 contributions to 274 benchmarks using 152 methods with more than 8 million data-points, and the leaderboard is continuously expanding. The JARVIS-Leaderboard is available at the website: https://pages.nist.gov/jarvis_leaderboard |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2306_11688 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | JARVIS-Leaderboard: A Large Scale Benchmark of Materials Design Methods Choudhary, Kamal Wines, Daniel Li, Kangming Garrity, Kevin F. Gupta, Vishu Romero, Aldo H. Krogel, Jaron T. Saritas, Kayahan Fuhr, Addis Ganesh, Panchapakesan Kent, Paul R. C. Yan, Keqiang Lin, Yuchao Ji, Shuiwang Blaiszik, Ben Reiser, Patrick Friederich, Pascal Agrawal, Ankit Tiwary, Pratyush Beyerle, Eric Minch, Peter Rhone, Trevor David Takeuchi, Ichiro Wexler, Robert B. Mannodi-Kanakkithodi, Arun Ertekin, Elif Mishra, Avanish Mathew, Nithin Baird, Sterling G. Wood, Mitchell Rohskopf, Andrew Dale Hattrick-Simpers, Jason Wang, Shih-Han Achenie, Luke E. K. Xin, Hongliang Williams, Maureen Biacchi, Adam J. Tavazza, Francesca Materials Science Lack of rigorous reproducibility and validation are major hurdles for scientific development across many fields. Materials science in particular encompasses a variety of experimental and theoretical approaches that require careful benchmarking. Leaderboard efforts have been developed previously to mitigate these issues. However, a comprehensive comparison and benchmarking on an integrated platform with multiple data modalities with both perfect and defect materials data is still lacking. This work introduces JARVIS-Leaderboard, an open-source and community-driven platform that facilitates benchmarking and enhances reproducibility. The platform allows users to set up benchmarks with custom tasks and enables contributions in the form of dataset, code, and meta-data submissions. We cover the following materials design categories: Artificial Intelligence (AI), Electronic Structure (ES), Force-fields (FF), Quantum Computation (QC) and Experiments (EXP). For AI, we cover several types of input data, including atomic structures, atomistic images, spectra, and text. For ES, we consider multiple ES approaches, software packages, pseudopotentials, materials, and properties, comparing results to experiment. For FF, we compare multiple approaches for material property predictions. For QC, we benchmark Hamiltonian simulations using various quantum algorithms and circuits. Finally, for experiments, we use the inter-laboratory approach to establish benchmarks. There are 1281 contributions to 274 benchmarks using 152 methods with more than 8 million data-points, and the leaderboard is continuously expanding. The JARVIS-Leaderboard is available at the website: https://pages.nist.gov/jarvis_leaderboard |
| title | JARVIS-Leaderboard: A Large Scale Benchmark of Materials Design Methods |
| topic | Materials Science |
| url | https://arxiv.org/abs/2306.11688 |