Auditing Google's AI Overviews and Featured Snippets: A Case Study on Baby Care and Pregnancy
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911533437550592 |
|---|---|
| author | Hu, Desheng Baumann, Joachim Urman, Aleksandra Lichtenegger, Elsa Forsberg, Robin Hannak, Aniko Wilson, Christo |
| author_facet | Hu, Desheng Baumann, Joachim Urman, Aleksandra Lichtenegger, Elsa Forsberg, Robin Hannak, Aniko Wilson, Christo |
| contents | Google Search increasingly surfaces AI-generated content through features like AI Overviews (AIO) and Featured Snippets (FS), which users frequently rely on despite having no control over their presentation. Through a systematic algorithm audit of 1,508 real baby care and pregnancy-related queries, we evaluate the quality and consistency of these information displays. Our robust evaluation framework assesses multiple quality dimensions, including answer consistency, relevance, presence of medical safeguards, source categories, and sentiment alignment. Our results reveal concerning gaps in information consistency, with information in AIO and FS displayed on the same search result page being inconsistent with each other in 33% of cases. Despite high relevance scores, both features critically lack medical safeguards (present in just 11% of AIO and 7% of FS responses). While health and wellness websites dominate source categories for both, AIO and FS, FS also often link to commercial sources. These findings have important implications for public health information access and demonstrate the need for stronger quality controls in AI-mediated health information. Our methodology provides a transferable framework for auditing AI systems across high-stakes domains where information quality directly impacts user well-being. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_12920 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Auditing Google's AI Overviews and Featured Snippets: A Case Study on Baby Care and Pregnancy Hu, Desheng Baumann, Joachim Urman, Aleksandra Lichtenegger, Elsa Forsberg, Robin Hannak, Aniko Wilson, Christo Computation and Language Artificial Intelligence Computers and Society Human-Computer Interaction Information Retrieval Google Search increasingly surfaces AI-generated content through features like AI Overviews (AIO) and Featured Snippets (FS), which users frequently rely on despite having no control over their presentation. Through a systematic algorithm audit of 1,508 real baby care and pregnancy-related queries, we evaluate the quality and consistency of these information displays. Our robust evaluation framework assesses multiple quality dimensions, including answer consistency, relevance, presence of medical safeguards, source categories, and sentiment alignment. Our results reveal concerning gaps in information consistency, with information in AIO and FS displayed on the same search result page being inconsistent with each other in 33% of cases. Despite high relevance scores, both features critically lack medical safeguards (present in just 11% of AIO and 7% of FS responses). While health and wellness websites dominate source categories for both, AIO and FS, FS also often link to commercial sources. These findings have important implications for public health information access and demonstrate the need for stronger quality controls in AI-mediated health information. Our methodology provides a transferable framework for auditing AI systems across high-stakes domains where information quality directly impacts user well-being. |
| title | Auditing Google's AI Overviews and Featured Snippets: A Case Study on Baby Care and Pregnancy |
| topic | Computation and Language Artificial Intelligence Computers and Society Human-Computer Interaction Information Retrieval |
| url | https://arxiv.org/abs/2511.12920 |