Can Consumer Chatbots Reason? A Student-Led Field Experiment Embedded in an "AI-for-All" Undergraduate Course
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917189064327168 |
|---|---|
| author | Shehu, Amarda Ababu, Adonyas Akbary, Asma Allen, Griffin Baig, Aroush Battle, Tereana Beall, Elias Byrom, Christopher Dean, Matt Demarco, Kate Douglass, Ethan Granados, Luis Hantush, Layla Hay, Andy Hay, Eleanor Jackson, Caleb Jang, Jaewon Jones, Carter Li, Quanyang Lopez, Adrian Massimo, Logan McMullin, Garrett Maldonado, Ariana Mendoza Mirza, Eman Muddasar, Hadiya Nuwayhid, Sara Pak, Brandon Petty, Ashley Rancourt, Dryden Rodriguez, Lily Rogers, Corbin Schiek, Jacob Seok, Taeseo Sethi, Aarav Vitela, Giovanni Williams, Winston Yetukuri, Jagan |
| author_facet | Shehu, Amarda Ababu, Adonyas Akbary, Asma Allen, Griffin Baig, Aroush Battle, Tereana Beall, Elias Byrom, Christopher Dean, Matt Demarco, Kate Douglass, Ethan Granados, Luis Hantush, Layla Hay, Andy Hay, Eleanor Jackson, Caleb Jang, Jaewon Jones, Carter Li, Quanyang Lopez, Adrian Massimo, Logan McMullin, Garrett Maldonado, Ariana Mendoza Mirza, Eman Muddasar, Hadiya Nuwayhid, Sara Pak, Brandon Petty, Ashley Rancourt, Dryden Rodriguez, Lily Rogers, Corbin Schiek, Jacob Seok, Taeseo Sethi, Aarav Vitela, Giovanni Williams, Winston Yetukuri, Jagan |
| contents | Claims about whether large language model (LLM) chatbots "reason" are typically debated using curated benchmarks and laboratory-style evaluation protocols. This paper offers a complementary perspective: a student-led field experiment embedded as a midterm project in UNIV 182 (AI4All) at George Mason University, a Mason Core course designed for undergraduates across disciplines with no expected prior STEM exposure. Student teams designed their own reasoning tasks, ran them on widely used consumer chatbots representative of current capabilities, and evaluated both (i) answer correctness and (ii) the validity of the chatbot's stated reasoning (for example, cases where an answer is correct but the explanation is not, or vice versa). Across eight teams that reported standardized scores, students contributed 80 original reasoning prompts spanning six categories: pattern completion, transformation rules, spatial/visual reasoning, quantitative reasoning, relational/logic reasoning, and analogical reasoning. These prompts yielded 320 model responses plus follow-up explanations. Aggregating team-level results, OpenAI GPT-5 and Claude 4.5 achieved the highest mean answer accuracy (86.2% and 83.8%), followed by Grok 4 (82.5%) and Perplexity (73.1%); explanation validity showed a similar ordering (81.2%, 80.0%, 77.5%, 66.2%). Qualitatively, teams converged on a consistent error signature: strong performance on short, structured math and pattern items but reduced reliability on spatial/visual reasoning and multi-step transformations, with frequent "sound right but reason wrong" explanations. The assignment's primary contribution is pedagogical: it operationalizes AI literacy as experimental practice (prompt design, measurement, rater disagreement, and interpretability/grounding) while producing a reusable, student-generated corpus of reasoning probes grounded in authentic end-user interaction. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_04225 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Can Consumer Chatbots Reason? A Student-Led Field Experiment Embedded in an "AI-for-All" Undergraduate Course Shehu, Amarda Ababu, Adonyas Akbary, Asma Allen, Griffin Baig, Aroush Battle, Tereana Beall, Elias Byrom, Christopher Dean, Matt Demarco, Kate Douglass, Ethan Granados, Luis Hantush, Layla Hay, Andy Hay, Eleanor Jackson, Caleb Jang, Jaewon Jones, Carter Li, Quanyang Lopez, Adrian Massimo, Logan McMullin, Garrett Maldonado, Ariana Mendoza Mirza, Eman Muddasar, Hadiya Nuwayhid, Sara Pak, Brandon Petty, Ashley Rancourt, Dryden Rodriguez, Lily Rogers, Corbin Schiek, Jacob Seok, Taeseo Sethi, Aarav Vitela, Giovanni Williams, Winston Yetukuri, Jagan Computers and Society Artificial Intelligence Claims about whether large language model (LLM) chatbots "reason" are typically debated using curated benchmarks and laboratory-style evaluation protocols. This paper offers a complementary perspective: a student-led field experiment embedded as a midterm project in UNIV 182 (AI4All) at George Mason University, a Mason Core course designed for undergraduates across disciplines with no expected prior STEM exposure. Student teams designed their own reasoning tasks, ran them on widely used consumer chatbots representative of current capabilities, and evaluated both (i) answer correctness and (ii) the validity of the chatbot's stated reasoning (for example, cases where an answer is correct but the explanation is not, or vice versa). Across eight teams that reported standardized scores, students contributed 80 original reasoning prompts spanning six categories: pattern completion, transformation rules, spatial/visual reasoning, quantitative reasoning, relational/logic reasoning, and analogical reasoning. These prompts yielded 320 model responses plus follow-up explanations. Aggregating team-level results, OpenAI GPT-5 and Claude 4.5 achieved the highest mean answer accuracy (86.2% and 83.8%), followed by Grok 4 (82.5%) and Perplexity (73.1%); explanation validity showed a similar ordering (81.2%, 80.0%, 77.5%, 66.2%). Qualitatively, teams converged on a consistent error signature: strong performance on short, structured math and pattern items but reduced reliability on spatial/visual reasoning and multi-step transformations, with frequent "sound right but reason wrong" explanations. The assignment's primary contribution is pedagogical: it operationalizes AI literacy as experimental practice (prompt design, measurement, rater disagreement, and interpretability/grounding) while producing a reusable, student-generated corpus of reasoning probes grounded in authentic end-user interaction. |
| title | Can Consumer Chatbots Reason? A Student-Led Field Experiment Embedded in an "AI-for-All" Undergraduate Course |
| topic | Computers and Society Artificial Intelligence |
| url | https://arxiv.org/abs/2601.04225 |