Consent in Crisis: The Rapid Decline of the AI Data Commons
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913443605381120 |
|---|---|
| author | Longpre, Shayne Mahari, Robert Lee, Ariel Lund, Campbell Oderinwale, Hamidah Brannon, William Saxena, Nayan Obeng-Marnu, Naana South, Tobin Hunter, Cole Klyman, Kevin Klamm, Christopher Schoelkopf, Hailey Singh, Nikhil Cherep, Manuel Anis, Ahmad Dinh, An Chitongo, Caroline Yin, Da Sileo, Damien Mataciunas, Deividas Misra, Diganta Alghamdi, Emad Shippole, Enrico Zhang, Jianguo Materzynska, Joanna Qian, Kun Tiwary, Kush Miranda, Lester Dey, Manan Liang, Minnie Hamdy, Mohammed Muennighoff, Niklas Ye, Seonghyeon Kim, Seungone Mohanty, Shrestha Gupta, Vipul Sharma, Vivek Chien, Vu Minh Zhou, Xuhui Li, Yizhi Xiong, Caiming Villa, Luis Biderman, Stella Li, Hanlin Ippolito, Daphne Hooker, Sara Kabbara, Jad Pentland, Sandy |
| author_facet | Longpre, Shayne Mahari, Robert Lee, Ariel Lund, Campbell Oderinwale, Hamidah Brannon, William Saxena, Nayan Obeng-Marnu, Naana South, Tobin Hunter, Cole Klyman, Kevin Klamm, Christopher Schoelkopf, Hailey Singh, Nikhil Cherep, Manuel Anis, Ahmad Dinh, An Chitongo, Caroline Yin, Da Sileo, Damien Mataciunas, Deividas Misra, Diganta Alghamdi, Emad Shippole, Enrico Zhang, Jianguo Materzynska, Joanna Qian, Kun Tiwary, Kush Miranda, Lester Dey, Manan Liang, Minnie Hamdy, Mohammed Muennighoff, Niklas Ye, Seonghyeon Kim, Seungone Mohanty, Shrestha Gupta, Vipul Sharma, Vivek Chien, Vu Minh Zhou, Xuhui Li, Yizhi Xiong, Caiming Villa, Luis Biderman, Stella Li, Hanlin Ippolito, Daphne Hooker, Sara Kabbara, Jad Pentland, Sandy |
| contents | General-purpose artificial intelligence (AI) systems are built on massive swathes of public web data, assembled into corpora such as C4, RefinedWeb, and Dolma. To our knowledge, we conduct the first, large-scale, longitudinal audit of the consent protocols for the web domains underlying AI training corpora. Our audit of 14,000 web domains provides an expansive view of crawlable web data and how codified data use preferences are changing over time. We observe a proliferation of AI-specific clauses to limit use, acute differences in restrictions on AI developers, as well as general inconsistencies between websites' expressed intentions in their Terms of Service and their robots.txt. We diagnose these as symptoms of ineffective web protocols, not designed to cope with the widespread re-purposing of the internet for AI. Our longitudinal analyses show that in a single year (2023-2024) there has been a rapid crescendo of data restrictions from web sources, rendering ~5%+ of all tokens in C4, or 28%+ of the most actively maintained, critical sources in C4, fully restricted from use. For Terms of Service crawling restrictions, a full 45% of C4 is now restricted. If respected or enforced, these restrictions are rapidly biasing the diversity, freshness, and scaling laws for general-purpose AI systems. We hope to illustrate the emerging crises in data consent, for both developers and creators. The foreclosure of much of the open web will impact not only commercial AI, but also non-commercial AI and academic research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_14933 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Consent in Crisis: The Rapid Decline of the AI Data Commons Longpre, Shayne Mahari, Robert Lee, Ariel Lund, Campbell Oderinwale, Hamidah Brannon, William Saxena, Nayan Obeng-Marnu, Naana South, Tobin Hunter, Cole Klyman, Kevin Klamm, Christopher Schoelkopf, Hailey Singh, Nikhil Cherep, Manuel Anis, Ahmad Dinh, An Chitongo, Caroline Yin, Da Sileo, Damien Mataciunas, Deividas Misra, Diganta Alghamdi, Emad Shippole, Enrico Zhang, Jianguo Materzynska, Joanna Qian, Kun Tiwary, Kush Miranda, Lester Dey, Manan Liang, Minnie Hamdy, Mohammed Muennighoff, Niklas Ye, Seonghyeon Kim, Seungone Mohanty, Shrestha Gupta, Vipul Sharma, Vivek Chien, Vu Minh Zhou, Xuhui Li, Yizhi Xiong, Caiming Villa, Luis Biderman, Stella Li, Hanlin Ippolito, Daphne Hooker, Sara Kabbara, Jad Pentland, Sandy Computation and Language Artificial Intelligence Machine Learning General-purpose artificial intelligence (AI) systems are built on massive swathes of public web data, assembled into corpora such as C4, RefinedWeb, and Dolma. To our knowledge, we conduct the first, large-scale, longitudinal audit of the consent protocols for the web domains underlying AI training corpora. Our audit of 14,000 web domains provides an expansive view of crawlable web data and how codified data use preferences are changing over time. We observe a proliferation of AI-specific clauses to limit use, acute differences in restrictions on AI developers, as well as general inconsistencies between websites' expressed intentions in their Terms of Service and their robots.txt. We diagnose these as symptoms of ineffective web protocols, not designed to cope with the widespread re-purposing of the internet for AI. Our longitudinal analyses show that in a single year (2023-2024) there has been a rapid crescendo of data restrictions from web sources, rendering ~5%+ of all tokens in C4, or 28%+ of the most actively maintained, critical sources in C4, fully restricted from use. For Terms of Service crawling restrictions, a full 45% of C4 is now restricted. If respected or enforced, these restrictions are rapidly biasing the diversity, freshness, and scaling laws for general-purpose AI systems. We hope to illustrate the emerging crises in data consent, for both developers and creators. The foreclosure of much of the open web will impact not only commercial AI, but also non-commercial AI and academic research. |
| title | Consent in Crisis: The Rapid Decline of the AI Data Commons |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2407.14933 |