The Responsible Foundation Model Development Cheatsheet: A Review of Tools & Resources
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910829598736384 |
|---|---|
| author | Longpre, Shayne Biderman, Stella Albalak, Alon Schoelkopf, Hailey McDuff, Daniel Kapoor, Sayash Klyman, Kevin Lo, Kyle Ilharco, Gabriel San, Nay Rauh, Maribeth Skowron, Aviya Vidgen, Bertie Weidinger, Laura Narayanan, Arvind Sanh, Victor Adelani, David Liang, Percy Bommasani, Rishi Henderson, Peter Luccioni, Sasha Jernite, Yacine Soldaini, Luca |
| author_facet | Longpre, Shayne Biderman, Stella Albalak, Alon Schoelkopf, Hailey McDuff, Daniel Kapoor, Sayash Klyman, Kevin Lo, Kyle Ilharco, Gabriel San, Nay Rauh, Maribeth Skowron, Aviya Vidgen, Bertie Weidinger, Laura Narayanan, Arvind Sanh, Victor Adelani, David Liang, Percy Bommasani, Rishi Henderson, Peter Luccioni, Sasha Jernite, Yacine Soldaini, Luca |
| contents | Foundation model development attracts a rapidly expanding body of contributors, scientists, and applications. To help shape responsible development practices, we introduce the Foundation Model Development Cheatsheet: a growing collection of 250+ tools and resources spanning text, vision, and speech modalities. We draw on a large body of prior work to survey resources (e.g. software, documentation, frameworks, guides, and practical tools) that support informed data selection, processing, and understanding, precise and limitation-aware artifact documentation, efficient model training, advance awareness of the environmental impact from training, careful model evaluation of capabilities, risks, and claims, as well as responsible model release, licensing and deployment practices. We hope this curated collection of resources helps guide more responsible development. The process of curating this list, enabled us to review the AI development ecosystem, revealing what tools are critically missing, misused, or over-used in existing practices. We find that (i) tools for data sourcing, model evaluation, and monitoring are critically under-serving ethical and real-world needs, (ii) evaluations for model safety, capabilities, and environmental impact all lack reproducibility and transparency, (iii) text and particularly English-centric analyses continue to dominate over multilingual and multi-modal analyses, and (iv) evaluation of systems, rather than just models, is needed so that capabilities and impact are assessed in context. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_16746 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | The Responsible Foundation Model Development Cheatsheet: A Review of Tools & Resources Longpre, Shayne Biderman, Stella Albalak, Alon Schoelkopf, Hailey McDuff, Daniel Kapoor, Sayash Klyman, Kevin Lo, Kyle Ilharco, Gabriel San, Nay Rauh, Maribeth Skowron, Aviya Vidgen, Bertie Weidinger, Laura Narayanan, Arvind Sanh, Victor Adelani, David Liang, Percy Bommasani, Rishi Henderson, Peter Luccioni, Sasha Jernite, Yacine Soldaini, Luca Machine Learning Artificial Intelligence Computation and Language Foundation model development attracts a rapidly expanding body of contributors, scientists, and applications. To help shape responsible development practices, we introduce the Foundation Model Development Cheatsheet: a growing collection of 250+ tools and resources spanning text, vision, and speech modalities. We draw on a large body of prior work to survey resources (e.g. software, documentation, frameworks, guides, and practical tools) that support informed data selection, processing, and understanding, precise and limitation-aware artifact documentation, efficient model training, advance awareness of the environmental impact from training, careful model evaluation of capabilities, risks, and claims, as well as responsible model release, licensing and deployment practices. We hope this curated collection of resources helps guide more responsible development. The process of curating this list, enabled us to review the AI development ecosystem, revealing what tools are critically missing, misused, or over-used in existing practices. We find that (i) tools for data sourcing, model evaluation, and monitoring are critically under-serving ethical and real-world needs, (ii) evaluations for model safety, capabilities, and environmental impact all lack reproducibility and transparency, (iii) text and particularly English-centric analyses continue to dominate over multilingual and multi-modal analyses, and (iv) evaluation of systems, rather than just models, is needed so that capabilities and impact are assessed in context. |
| title | The Responsible Foundation Model Development Cheatsheet: A Review of Tools & Resources |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2406.16746 |