Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2402.07911 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909871756017664 |
|---|---|
| author | Walton, Sean P. Evans, Ben J. Rahat, Alma A. M. Stovold, James Vincalek, Jakub |
| author_facet | Walton, Sean P. Evans, Ben J. Rahat, Alma A. M. Stovold, James Vincalek, Jakub |
| contents | As AI systems increasingly shape decision making in creative design contexts, understanding how humans engage with these tools has become a critical challenge for interactive intelligent systems research. This paper contributes a challenge to rethink how to evaluate human--AI collaborative systems, advocating for a more nuanced and multidimensional approach. Findings from one of the largest field studies to date (n = 808) of a human--AI co-creative system, The Genetic Car Designer, complemented by a controlled lab study (n = 12) are presented. The system is based on an interactive evolutionary algorithm where participants were tasked with designing a simple two dimensional representation of a car. Participants were exposed to galleries of design suggestions generated by an intelligent system, MAP--Elites, and a random control. Results indicate that exposure to galleries generated by MAP--Elites significantly enhanced both cognitive and behavioural engagement, leading to higher-quality design outcomes. Crucially for the wider community, the analysis reveals that conventional evaluation methods, which often focus on solely behavioural and design quality metrics, fail to capture the full spectrum of user engagement. By considering the human--AI design process as a changing emotional, behavioural and cognitive state of the designer, we propose evaluating human--AI systems holistically and considering intelligent systems as a core part of the user experience -- not simply a back end tool. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_07911 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | From Metrics to Meaning: Time to Rethink Evaluation in Human-AI Collaborative Design Walton, Sean P. Evans, Ben J. Rahat, Alma A. M. Stovold, James Vincalek, Jakub Human-Computer Interaction Artificial Intelligence Computational Engineering, Finance, and Science Neural and Evolutionary Computing I.2.0; J.6; G.1.6 As AI systems increasingly shape decision making in creative design contexts, understanding how humans engage with these tools has become a critical challenge for interactive intelligent systems research. This paper contributes a challenge to rethink how to evaluate human--AI collaborative systems, advocating for a more nuanced and multidimensional approach. Findings from one of the largest field studies to date (n = 808) of a human--AI co-creative system, The Genetic Car Designer, complemented by a controlled lab study (n = 12) are presented. The system is based on an interactive evolutionary algorithm where participants were tasked with designing a simple two dimensional representation of a car. Participants were exposed to galleries of design suggestions generated by an intelligent system, MAP--Elites, and a random control. Results indicate that exposure to galleries generated by MAP--Elites significantly enhanced both cognitive and behavioural engagement, leading to higher-quality design outcomes. Crucially for the wider community, the analysis reveals that conventional evaluation methods, which often focus on solely behavioural and design quality metrics, fail to capture the full spectrum of user engagement. By considering the human--AI design process as a changing emotional, behavioural and cognitive state of the designer, we propose evaluating human--AI systems holistically and considering intelligent systems as a core part of the user experience -- not simply a back end tool. |
| title | From Metrics to Meaning: Time to Rethink Evaluation in Human-AI Collaborative Design |
| topic | Human-Computer Interaction Artificial Intelligence Computational Engineering, Finance, and Science Neural and Evolutionary Computing I.2.0; J.6; G.1.6 |
| url | https://arxiv.org/abs/2402.07911 |